When two computers, say X and Y establish a TCP/IP connection, the initial transaction, known as a three-way handshake, is something like this. (Note: SYN means Synchronization packet while ACK means Acknowledgement packet).
X sends a SYN packet to Y. Y replies with a SYN ACK packet. X then replies with an ACK packet.
Only upon completion of these steps is the TCP/IP connection established.
In a SYN attack, numerous SYN packets, all having bad source IP addresses (e.g. non-existent IP address), are sent to the target machine. The target machine responds to each SYN packet with a SYN ACK packet and waits for the ACK packet. Since the source IP address in all these is bad, the ACK packet never comes. Meanwhile the requests get added to the queue resulting in blockage of resources. A sufficiently large number of such SYN packets can crash / hang / reboot the system.