Sockets Distributed Systems Philipp Kupferschmied Universita¨t Karlsruhe, System Architecture Group May 6th, 2009 Philipp Kupferschmied Sockets 1/ 16 The Socket Abstraction User’s Point of View Sockets in Linux 1 The Socket Abstraction Socket Basics Characteristics 2 User’s Point of View Connections Single-Threaded Servers Summary 3 Sockets in Linux Data Structures Traversing Layers Conclusion Philipp Kupferschmied Sockets 2/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Socket Basics Characteristics What are Sockets? Socket Socket Kernel Process A Process B Connection oriented IPC “Phone connection” between processes Protocol determines how the connection is used Philipp Kupferschmied Sockets 3/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Socket Basics Characteristics Socket Types Socket File descriptor (sock fd) Protocol family Unix-domain (AF UNIX) Internet-domain (AF INET) Socket type Stream (SOCK STREAM) Datagram (SOCK DGRAM) Protocol UNIX STREAM, UNIX DGRAM TCP, UDP Philipp Kupferschmied Sockets 4/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Socket Basics Characteristics Socket Types Socket File descriptor (sock fd) Protocol family Unix-domain (AF UNIX) Internet-domain (AF INET) Socket type Stream (SOCK STREAM) Datagram (SOCK DGRAM) Protocol UNIX STREAM, UNIX DGRAM TCP, UDP Philipp Kupferschmied Sockets 4/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Socket Basics Characteristics Socket Types Socket File descriptor (sock fd) Protocol family Unix-domain (AF UNIX) Internet-domain (AF INET) Socket type Stream (SOCK STREAM) Datagram (SOCK DGRAM) Protocol UNIX STREAM, UNIX DGRAM TCP, UDP Philipp Kupferschmied Sockets 4/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Socket Basics Characteristics Socket Types Socket File descriptor (sock fd) Protocol family Unix-domain (AF UNIX) Internet-domain (AF INET) Socket type Stream (SOCK STREAM) Datagram (SOCK DGRAM) Protocol UNIX STREAM, UNIX DGRAM TCP, UDP Philipp Kupferschmied Sockets 4/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Socket Basics Characteristics Unix-Domain Sockets Implemented purely in memory Users can bind a socket to an “address” Socket becomes accessible at this “address” Unix-domain addresses are file names /tmp/server sockaddr un sun family = AF UNIX sun path = /tmp/server Socket Socket Client Server Philipp Kupferschmied Sockets 5/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Socket Basics Characteristics Unix-Domain Sockets Implemented purely in memory Users can bind a socket to an “address” Socket becomes accessible at this “address” Unix-domain addresses are file names /tmp/server sockaddr un sun family = AF UNIX sun path = /tmp/server Socket Socket Client Server / tmp/server Philipp Kupferschmied Sockets 5/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Socket Basics Characteristics Unix-Domain Sockets Implemented purely in memory Users can bind a socket to an “address” Socket becomes accessible at this “address” Unix-domain addresses are file names /tmp/server sockaddr un sun family = AF UNIX sun path = /tmp/server Socket Socket Client Server / tmp/server request Philipp Kupferschmied Sockets 5/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Socket Basics Characteristics Unix-Domain Sockets Implemented purely in memory Users can bind a socket to an “address” Socket becomes accessible at this “address” Unix-domain addresses are file names /tmp/server sockaddr un sun family = AF UNIX sun path = /tmp/server Socket Socket Client Server / tmp/server request reply Philipp Kupferschmied Sockets 5/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Socket Basics Characteristics Internet-Domain Sockets Implemented via the network Users can bind a socket to an “address” Socket becomes accessible at this “address” Internet-domain addresses consist of IP and port number 192.168.0.1:80 sockaddr in sin family = AF INET sin port = 80 sin addr = 0x0100A8C0 Socket Socket Client Server Philipp Kupferschmied Sockets 6/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Socket Basics Characteristics Internet-Domain Sockets Implemented via the network Users can bind a socket to an “address” Socket becomes accessible at this “address” Internet-domain addresses consist of IP and port number 192.168.0.1:80 sockaddr in sin family = AF INET sin port = 80 sin addr = 0x0100A8C0 Socket Socket Client Server 192.168.0.1:80192.168.0.1:3245 Philipp Kupferschmied Sockets 6/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Socket Basics Characteristics Internet-Domain Sockets Implemented via the network Users can bind a socket to an “address” Socket becomes accessible at this “address” Internet-domain addresses consist of IP and port number 192.168.0.1:80 sockaddr in sin family = AF INET sin port = 80 sin addr = 0x0100A8C0 Socket Socket Client Server 192.168.0.1:80 request 192.168.0.1:3245 Philipp Kupferschmied Sockets 6/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Socket Basics Characteristics Internet-Domain Sockets Implemented via the network Users can bind a socket to an “address” Socket becomes accessible at this “address” Internet-domain addresses consist of IP and port number 192.168.0.1:80 sockaddr in sin family = AF INET sin port = 80 sin addr = 0x0100A8C0 Socket Socket Client Server 192.168.0.1:80 request reply 192.168.0.1:3245 Philipp Kupferschmied Sockets 6/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Socket Basics Characteristics Characteristics Communication between two sockets Connection-oriented or Packet-based Allow for error control (TCP, UNIX STREAM) Network-capable IPC Flexible AX 25, APPLETALK, IPX, HTTP, RPC, . . . Philipp Kupferschmied Sockets 7/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Connections Single-Threaded Servers Summary Establishing a Connection socket() socket Philipp Kupferschmied Sockets 8/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Connections Single-Threaded Servers Summary Establishing a Connection bind() socket Philipp Kupferschmied Sockets 8/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Connections Single-Threaded Servers Summary Establishing a Connection l isten() socket Philipp Kupferschmied Sockets 8/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Connections Single-Threaded Servers Summary Establishing a Connection accept() socket Philipp Kupferschmied Sockets 8/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Connections Single-Threaded Servers Summary Establishing a Connection accept() socket()socket socket Philipp Kupferschmied Sockets 8/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Connections Single-Threaded Servers Summary Establishing a Connection accept() connect()socketsocket Philipp Kupferschmied Sockets 8/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Connections Single-Threaded Servers Summary Establishing a Connection socket’ socket socket read() wri te() close() wri te() read() close() Philipp Kupferschmied Sockets 8/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Connections Single-Threaded Servers Summary Establishing a Connection accept() socket Philipp Kupferschmied Sockets 8/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Connections Single-Threaded Servers Summary Establishing a Connection socket() socket bind() l isten() accept() connect() socket()socket socket socket socket’ socket socket read() wri te() close() wri te() read() close() socket Philipp Kupferschmied Sockets 8/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Connections Single-Threaded Servers Summary Data Transfer socket’ socketread() wri te() wri te() read() socket socket’ socket’’ socket socket socket connect() wri te() wri te()read() read() accept() Blocking syscalls ‘accept’ and ‘read’ require a thread per socket. Philipp Kupferschmied Sockets 9/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Connections Single-Threaded Servers Summary Data Transfer socket’ socketread() wri te() wri te() read() socket socket’ socket’’ socket socket socket connect() wri te() wri te() fd_set select() or poll() Use ‘select’ or ‘poll’ to watch many sockets in one thread. Philipp Kupferschmied Sockets 9/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Connections Single-Threaded Servers Summary Strengths and Weaknesses + Encapsulation of data transfer + Extensibility with new protocols + Flexibility (e.g., X server) Unix-domain sockets for local clients Internet-domain sockets for remote clients Same code, except connect() – Inconvenient programming interface – Each process has to close its local socket Philipp Kupferschmied Sockets 10/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Connections Single-Threaded Servers Summary Example (Unix-Domain) Client: int s = socket ( AF_UNIX, SOCK_STREAM, 0 ); struct sockaddr_un server_addr; server_addr.sun_family = AF_UNIX; strcpy( server_addr.sun_path, "/tmp/serversock" ); connect( s, (struct sockaddr *) &server_addr, sizeof(server_addr) ); Server: s = socket( AF_UNIX, SOCK_STREAM, 0 ); struct sockaddr_un server_addr; server_addr.sun_family = AF_UNIX; strcpy( server_addr.sun_path, "/tmp/serversock" ); bind( s, (struct sockaddr *)&server_addr, sizeof(server_addr) ); listen( s, 5 ); Philipp Kupferschmied Sockets 11/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Connections Single-Threaded Servers Summary Example (Internet-Domain) Client: int s = socket( AF_INET, SOCK_STREAM, 0 ); struct sockaddr_in server_addr; server_addr.sin_family = AF_INET; server_addr.sin_port = htons(15001); inet_aton( "127.0.0.1", &server_addr.sin_addr ); connect( s, (struct sockaddr *)&server_addr, sizeof(server_addr) ); Server: int s = socket( AF_INET, SOCK_STREAM, 0 ); struct sockaddr_in server_addr; server_addr.sin_family = AF_INET; server_addr.sin_port = htons(15001); inet_aton( "127.0.0.1", &server_addr.sin_addr ); bind( s, (struct sockaddr *)&server_addr, sizeof(server_addr) ); listen( s, 5 ); Philipp Kupferschmied Sockets 12/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Data Structures Traversing Layers Conclusion struct socket struct socket state flags proto ops fasync list file sk wait type struct proto ops inet write inet read . . . struct sock sock common sk protocol sk type . . . head tail struct sk buff next prev sk dev data . . . Philipp Kupferschmied Sockets 13/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Data Structures Traversing Layers Conclusion struct socket struct socket state flags proto ops fasync list file sk wait type struct proto ops inet write inet read . . . struct sock sock common sk protocol sk type . . . head tail struct sk buff next prev sk dev data . . . Philipp Kupferschmied Sockets 13/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Data Structures Traversing Layers Conclusion struct socket struct socket state flags proto ops fasync list file sk wait type struct proto ops inet write inet read . . . struct sock sock common sk protocol sk type . . . head tail struct sk buff next prev sk dev data . . . Philipp Kupferschmied Sockets 13/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Data Structures Traversing Layers Conclusion struct socket struct socket state flags proto ops fasync list file sk wait type struct proto ops inet write inet read . . . struct sock sock common sk protocol sk type . . . head tail struct sk buff next prev sk dev data . . . Philipp Kupferschmied Sockets 13/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Data Structures Traversing Layers Conclusion Linux Sockets in Action write(fd) sys_write(fd) sock_write() inet_sendmsg() tcp_sendmsg() tcp_send_skb() tcp_transmit_skb() ip_queue_xmit() ei_start_xmit() read() sys_read() sock_read() inet_recvmsg() tcp_recvmsg() tcp_data() tcp_rcv() ip_recv() ei_recv ei_interrupt proto_ops data USER VFS TRANSPORT NETWORK DATA LINK PHYSICAL socket sock data data Philipp Kupferschmied Sockets 14/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Data Structures Traversing Layers Conclusion Linux Sockets in Action write(fd) sys_write(fd) sock_write() inet_sendmsg() tcp_sendmsg() tcp_send_skb() tcp_transmit_skb() ip_queue_xmit() ei_start_xmit() read(fd) sys_read(fd) sock_read() inet_recvmsg() tcp_recvmsg() tcp_data() tcp_rcv() ip_recv() ei_recv ei_interrupt proto_ops data USER VFS TRANSPORT NETWORK DATA LINK PHYSICAL socket sock data Philipp Kupferschmied Sockets 14/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Data Structures Traversing Layers Conclusion Linux Sockets in Action write(fd) sys_write(fd) sock_write() inet_sendmsg() tcp_sendmsg() tcp_send_skb() tcp_transmit_skb() ip_queue_xmit() ei_start_xmit() read(fd) sys_read(fd) sock_read() inet_recvmsg() tcp_recvmsg() tcp_data() tcp_rcv() ip_recv() ei_recv ei_interrupt proto_ops data USER VFS TRANSPORT NETWORK DATA LINK PHYSICAL socket sock data data Philipp Kupferschmied Sockets 14/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Data Structures Traversing Layers Conclusion Linux Sockets in Action write(fd) sys_write(fd) sock_write() inet_sendmsg() tcp_sendmsg() tcp_send_skb() tcp_transmit_skb() ip_queue_xmit() ei_start_xmit() read(fd) sys_read(fd) sock_read() inet_recvmsg() tcp_recvmsg() tcp_data() tcp_rcv() ip_recv() ei_recv ei_interrupt proto_ops data USER VFS TRANSPORT NETWORK DATA LINK PHYSICAL socket sock data Philipp Kupferschmied Sockets 14/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Data Structures Traversing Layers Conclusion Conclusions Sockets are an abstract communication interface Hide different protocols (Unix-domain vs. Internet-domain) Protocol dictates communication model Transient vs. persistent Unix’ish “everything is a file” paradigm Linux implementations are similar for different protocols Realized via function pointers and matching data structures Homebrew inheritance, “object-oriented C” (Questionable) solution to both performance and structured software Philipp Kupferschmied Sockets 15/ 16 The Socket Abstraction User’s Point of View Sockets in Linux Data Structures Traversing Layers Conclusion Literature Sources of the Linux kernel 2.4/2.6 Helmut Herold: Linux/Unix Systemprogrammierung Ju¨rgen Wolf: Linux-Unix-Programmierung http://www.pronix.de/pronix-6.html Beck: Linux Kernelprogrammierung Online Tutorial: http://www.cs.rpi.edu/academics/courses/netprog/ Programming Assignment You can gain experience in socket programming by implementing RPC via sockets in the programming assignment! Philipp Kupferschmied Sockets 16/ 16