Bereishis
Perek 4 Pesukim 2 - 4
ותסף ללדת את אחיו את הבל ויהי הבל רעה צאן וקין היה עבד אדמה
ויהי מקץ ימים ויבא קין מפרי האדמה מנחה להשם
והבל הביא גם הוא מבכֹרות צאנו ומחלבהן וישע ה' אל הבל ואל מנחתו
ואל קין ואל מנחתו לא שעה ויחר לקין מאד ויפלו פניו
The Torah does not make clear why Kayin's offering was not accepted. However, there are two inferences as to where he went wrong.
Regarding Kayin's offering, it is not written that he brought from the first and the best of his fruit as it does regarding Hevel's offering. We can infer that Kayin did not offer his best. As the Midrashim explains that Kayin brought the worst of his fruits[1] or his leftovers.[2] This is one possible issue.
R. Yitzchak Abarbanel rejects this explanation because Chazal taught us[3]
אחד המרבה ואחד הממעיט ובלבד שיכוין לבו לשמים.
It is not the quality or quantity of the Korban that counts; it is what is in the heart.
The second inference is that the Torah tells us that Hevel was a shepherd while Kayin worked the land. Some commentators explain that it was this choice of occupation that made all the difference. After Adam sinned the land was cursed. Farming is labor-intensive and time-consuming. Shepherding is not. A shepherd has time to focus on other aspects of life. In addition, a shepherd learns to be sensitive to other beings. For this reason, the Avos as well as Moshe and David were shepherds.[4] Kayin had been following the way of his father by working the land which had been cursed. He saw that Hevel became a shepherd and was living a better life yet he continued to be a farmer. The passuk first states that Hevel became a shepherd and then says that Kayin was a farmer. It would have made more sense to write about Kayin first as he was the firstborn. The explanation may be that Kayin's occupation did not need to be stated because he was simply continuing what his father had been doing. The meaning of the passuk is that Hevel chose to be a shepherd and yet Kayin remained a farmer.[5] And that was why Kayin's offering was not accepted. The lesson is that Hashem wants people to look for ways to improve their lives.[6]
The Torah writes:
"And Hashem turned to Hevel and his offering but to Kayin and his offering He did not turn".
It seems to be telling us that both of the above-mentioned issues were true. Kayin was rejected for his lifestyle choice and his offering was rejected because of its inferior quality.
The Meshech Chochma gives a different explanation. The Torah (Vayikra 2:11) writes that it is prohibited to sacrifice date honey on the mizbeach. The reason is that Hashem only wants korbanos from things that people worked with like animals that take work to raise and care for. This is why an animal cannot be brought until it is at least eight days old. Fruits and vegetables grow naturally without human effort so it is not desirable as a korban.[7] Only flour and wine that are man-made may be sacrificed.[8] This is why the animal sacrifice of Hevel was accepted and the vegetable sacrifice of Kayin was not. The lesson: that which takes effort is what is desirable to Hashem.
Fun fact:
The Midrash says that the offering of Kayin was flax seeds. After the incident with Kayin and Hevel Hashem said "the offering of Kayin shall never mix with the offering of Hevel". This is the reason for the issur of shatnez.[9]
[1] Bereshis Rabah 22:5.
[2] Tanchuma 9, Pirkei D'rabbi Elazar 21.
[3] Menachos 110a.
[4] R. Hirsch. He also explains how farming culture leads to idol worship and slavery as we see happened in Mitzrayim. (They also considered shepherding, the opposing worldview, an abomination.) The Torah gives many mitzvos relating to farming in order to avoid this potential danger.
[5] Ha'amek Davar.
[6] The Netziv in Ha'amek Davar explains that farming produces bread, an essential food. Sheep provide wool and milk which are not necessities (it wasn't permitted to eat meat yet). Kayin thought that the correct path in life is to live with the bare minimum that one needs to live. Hevel argued that one should try to advance the world and enjoy life as best as one could. The lesson is that Hevel's outlook is correct.
Yoram Hazony in The Philosophy of Hebrew Scripture writes:
"Cain has piously accepted the curse on the soil, and God’s having sent Adam to work the soil, as unchallengeable. His response is to submit, as his father did before him. And within the framework of this submission, he initiates ways of giving up of what little he has as an offer of thanksgiving. . . . Cain represents the life of the farmer, a life of pious submission, obeying in gratitude the custom that has been handed down, which alone provides bread so that man may live. Abel takes the curse on the soil as a fact, but not as one that possesses any intrinsic merit, so that it should command his allegiance. The fact that God has decreed it, and that his father has submitted to it, does not make it good. His response is the opposite of submission: He resists with ingenuity and daring, risking the anger of man and God to secure improvement for himself and for his children. Abel represents the life of the shepherd, which is a life of dissent and initiative, whose aim is to find the good life for man, which is presumed to be God’s true will."
[7] Farming takes a lot of work; however, the work is done before the fruit grows. The item being sacrificed should be something that was worked with.
[8] Although date honey is made by man, it is considered to be a downgrade from the dates themselves and therefore undesirable as a korban.
[9] Tanchuma 9, Pirkei D'rabbi Elazar 21.