1. Milestone 2
1.0. Deprecated Project Notice
20050122 Warning: This project is indefinitely put on hold.
20050225 Update: This is a utility library. It only advances when some other project needs this library. Right now I am working on some other projects that are [...to be continued]
1.1. Introduction
1.1.1. Summary
- Title: Simple Network Exchange II
- Acronym: SNE 2
- Goal: A simplified network layer library; using .Net socket programming.
- Language: C#. Python.
- Targets: VS.Net, Mono
- License: LGPL
- Started: 2004/11/04
1.1.2. Goal & Description
SNE is a rather simple library which purpose is to provide an easy way to build applications that need to exchange network information.
The library does not intend to be a comprehensive network library. Using network sockets under .Net is relatively easy -- most of the work is done by classes such as TcpClient and TcpListenner. Yet adding network-based interaction in an application requires a non-minimal amount of work which turns out to be pretty much generic. SNE tries to fill this gap by providing a simple skeletton on which network interaction can be based.
The keyword is "simple" here.
As such it defines a network-based application as two entities, the server and the clients. A server waits for TCP connections on a given port/address and serves services. Clients connect to a given port, address and service name. One the connection is established, server and clients exchange binary packets. The library does not impose any behavior or control on the nature of the protocol exchange between a client and a server. It is up to the library users to define their own protocol.
The notion of "services" has been introduced in an effort to further simplify usage implementation. Alternate names describing the concept would be "channels" or "protocols". For example a mail server could offer three services: one to retrieve the list of users, one to retrieve the list of folders from a given user and eventually one to retrieve the mail for a given user. The same mail server implementation could just decide to use only one service and move the distinction at the level of their own private protocol.
Implementation-wise, what is needed?
- A server needs to provide one connection-handling class per service. One instance of this class is created for every client connection on that service and is responsible for reading incoming packets sent by the client (and of course reply as needed by the service's protocol.)
- A client just needs to connect to a server and can then send and wait for packets synchronously. Optionally the client can set the receiver in asynchronous mode and be notified by a delegate when a packet is received.
- The host (i.e. the user of the library) is fully responsible of defining the protocol exchange. To simplify this task, all communication is done by exchaning "packets", i.e. byte buffers of arbitrary length. The default implementation provides an easy mechanism to treat these buffers as Unicode strings. OTOH a client could choose to exchange XML data simply by serializing it into the packets' data buffers.
Finally note that the SNE 2 library is been implemented on two different platforms: Python and .Net Compact Framework 1.1.
.Net 2.0 is too young at this time to be actually deployed. Usage of the .Net CF allows the library to be used on Pocket PC applications at the cost of a limited API for socket programming.
1.2. Design & Spec
1.3. Milestones
20041120
1.4. Plan
To be done (by decreasing priority):
2004MMDD [1.N] Section: Desc
Finished (by decreasing date):
2004MMDD [1.N] Section: Desc
1.5. Notes
(Notes are given in chronological order.)
- Concept: one server, N clients. Server defines "services" (Int or name) with a RConnection derived class type to instanciate. Client connects to server for a given service then exchange RPackets (Size + bulk)
- Server calls delegate when cnx opens or close.
- RServer: started by desktop, N connections, one only in debug
- RConnection server side : Client or server ip, connection index, abstract receive / send RPacket.
- RClient: connects to one server (ip, port, service name). Same RConnection-derived class idea with delegates.
- Server side compatible with .Net fx 1.1/2.0 and Mono (no generics).
- Client side compatible with .Net compact fx 1.1/2.0 (no generics).
An overview of SNE M2:
There's one Server and multiple Clients connect to that server on a given port and address.
The server provides multiple services. Each service has a different name. A client connects to one service on one server.
On the server side, a service is matched by a RServiceConnection class. That is when the server is created, the host defines each service by providing a name and a class that will match incoming connections. When a client connects to a service, the server instantiate one instance of the RServiceConnection class and provides it one user parameter.
RServiceConnection derives from RConnection which provide base attributes:
- A Send method.
- A Receive method.
- The IP of the end point connected to the service.
The client and the server exchange RPackets, a packed array of bytes with a given size. SNE M2 does not provide nor enforce any semantics for the packet itself, it is purely service dependant.
A client works differently. It instantiates a RClientConnection and connects to a server on a given port, address and service name. Once the connection is established, the client can use Send to send a packet and can use Receive the get a packet. A client can use three patterns:
- Use an instance of RClientConnection in "synchronous" mode, that is use the Send method then use the blocking version of Receive.
- Use an instance of RClientConnection in "asynchronous" mode, that is use the Send method. Received packets are queued and the client can access the queue when needed to retrieve packets. Optionaly the client can be notified by a delegate callback that a packet was received.
- Derive from RClientConnection and override the Receive method to perform processing as soon as a packet is received.
The exchange protocol between the server and the client is very basic. When a client connects, it first sends two strings of fixed length:
- String Signature: "alfray.sne" (10 bytes)
All strings are pure 8-bit UTF-8 strings. Valid characters are anything except control characters (< 20, space) and '\n'.
After that the exchange between the client and the server is based on requests and responses. A request has a string header in the form:
"x:data\n"where "x" is a letter naming the request and data is a parameter for that request. The line ends with a '\n' character.
There's an arbitrary limit of 1024 characters in the request line.
Once a request is sent by, the receiver replies with a response code. This status code is encoded as a string in the form:
"r:NNN\n"where "r" stands for "response" and NNN are 3 digits forming a number similar to the HTTP response codes. Typical examples are 200 (OK), 400 (Bad request), 404 (Not found), etc.
So after the signature has been accepted, the client must send its version number:
"v:10\n"which stands for version "1.0".
The server will reply with either:
- 200 (OK). Version accepted.
- 101 (Protocol differs).
- 505 (Version not supported).
If the server replied 200, it will consequently accept incoming packets.
If not, it will close the connection.
101 is for future extension and may result in protocol negotiation.
101 is not currently supported.
After that the client specifies the service name:
"s:<service name>\n"
The server replies to the service name request using on of the following responde codes:
- 200 (OK). The service exists. The server will accept packets.
- 404 (Not found). Service name does not exist.
If the server replied 200, it will consequently accept incoming packets. If not, it will close the connection.
The exchange of a packet is quite straightforward.
The sender sends the packer header as a string
"p:bNNN\n"where NNN is the size of the packets to follow in bytes. Then the bytes are send as-is.
There are currently no flags, no CRC, no transmit window and no packet index.
It is expected that in the future the packet exchange may be expanded with such features. Typically CRC and re-transmit would be nice. To do this, the packet header may look like this:
"p:bNNN:cNNN:iNNN".
Each flag is composed of a colon separator and a letter. The number being is the decimal value for the flag (it may be more compact to transmit the CRC as hexa, yet more generic to transmit all values the same way.) The index will be incremented for each packet and can be used to retransmit the same packet, with the same index.
Note that since SNE M2 solely operates on TCP connected streams, error detection and retransmit is already handled by TCP and it is thus not expected that an extension to the packet header actually be that useful.
Whenever a sender sends a packet, the receiver replies with a response code:
- 200 (OK). Packet accepted.
- 400 (Bad request). Packet was refused.
Error 400 has a loose meaning here. Since of the TCP nature of the connection, it is not expected that the packet was not correctly received unless the receiver is in a bad shape. It is up to the user of the protocol to decide what to do if this situation arises.
Note that the protocol does not offer compression yet. It may be added later as a flag:
"p:bNNN:zNNN"where "b" indicates the compressed size and "z" the uncompressed size, as well as the compression method (each one will use a different letter, "z" meaning zlib). CRC, if any, would be added on the compressed data.
I need a Python implementation to run it under Linux, instead of a Mono implementation. The footprint of Mono is still too big. That and Python is perfectly suited for the task.
An "alternative" implementation would be to simply rely on HTTP for the transport and the wire protocol. The benefit is obvious: passing thru firewalls is a lot easier.
Implementation wise, System.Net.HttpWebRequest is available in .Net CF 1.1. The .Net Framework does not seem to have an HTTP server just as-is. The only think I see for the server part is HttpServerChannel in System.Runtime.Remoting, which is not available on .Net CF 1.1 anyway. Well since CF doesn't have remoting, I wasn't really expecting an HTTP-server-in-one-class implementation to fall of the sky just like that :-)
Now, what does it take to simply "mimic" an HTTP protocol on the wire? Like a very basic server and client that does only the basic part HTTP 1.0? Just enough to go thru proxies and firewalls.
Obviously still a lot more than the protocol shown above.
As of November 2004, this project is officially abandonned._
Development has been moved into SNE M2.
Summary of this project:
Trivia: Originally I wanted to call this library SNS2 or SNL.
Simple Network Server version 1, refered as SNS, was supposedly a .Net C# class library
which would deliver a simple message-oriented network layer for developing games. It was not
intended to be the ultimate network layer nor anything like that; the goal was merely to explore
socket programming under .Net.
SNS never made it (the optimist I am would add yet).
The only goal which has been reached is socket programming under .Net,
knowledge that I reused later at work, and then extended to Java and WinSock and regular
unix sockets under linux. So it's been useful as a "refreshing" tool.
Now it's time to deal with the initial purpose of SNS, a simplified message-oriented network layer.
The SNE project can be seen as a version 2 of SNS.
It differs in the fact that it drops the "for developing games" part of
the initial goal. The purpose here is to provide a mechanism for two clients to connect and
exchange messages, ideally with as less code in the client as possible.
Additionnaly, the semantic of what a message is differs. In SNS the idea was to have a server
that would hold a "game state" and basically act as a message board. Clients would connect to
an instance of a game on the server and modify this state or be notified when the state changes.
This semantic is dropped in SNE.
On the design side, as in all protocols or file formats I've ever designed, there will be
some capabilities and versionning built in. Capabilities are my way of extending a protocol
without breaking everything (doesn't mean it has to be backward compatible; the descent scenario
is to know why it is not compatible). Capabilities I envision are compression, interopability
and selection of the serializatiom method.
An idea is to be able to go thru firewalls by using port 80 and non-binary data.
I believe I need a goal to keep working in the real stuff beside the initial spec.
One of the main design goals is for the library to be as easy to use as possible. That's mostly
because I'm so lazy that if it requires more than 2 lines of codes I'll be already thinking of
another phantom project instead.
Note that SNS could be implemented on top of SNE. Rather ironic.
Oh and the whole stuff will be LGPL.
Yes I like milestones. These will change with time. In fact pretty must everything below
can be considered as highly volatile and unreliable except the milestone 1.
So an initial tentative milestone list would be:
20031223
This should be a pure peer-to-peer communication library.
I want at least this feature set:
20031223
These are the details I envision for the protocol:
Workflow for initiation and protocol negotiation:
Data exchange will be described later.
20040101
This describes how I envision SNE being used from the host point of view.
There are two scenarios. I changed my mind since above. The "symmetric" non-server idea doesn't seem to fit my needs in
the first place. For simplicity, it is still easier to have a server and clients connecting to the server. The server
can be defined as a TCP listening process: an entity that waits for connections. Once a connection is established, then
SNE words as a peer-to-peer "symmetric" echange of data.
First, let's examine the "client" point of view. The client is the one that arbitrarily decides to start a connection
to a given known server. A host would use the SNE client like this:
Internally, the SNE client will probably send messages synchronously and keep a queue of received messages.
The definition of a message is up to both hosts communicating.
If a reply is needed for a specific sent message, it is up to the host to handle that.
Two simples ways to handle it would be: read any received messages immediately after sending one till the receive queue is
empty; or place an incremental index in the message data and only accept a reply with the same index.
Alternatively, I'd like a client to be able to register a MessageReceivedDelegate. The delegate would be called each time
a message has been received. I would have to make sure I use PInvoke or something to call the host in its main thread (rather
than from SNE's reception thread). Another approach would be to raise a signal. Another approach would be to have a specific
thread in the SNE client which task is to call the host delegate. Each message could then be processed directly, without
blocking the reception thread and without handling all messages in parallel (I was also thinking each time a message is
ready the host delegate could be called in a new thread, but then the host would experience a massive threading syndrome).
I'll take the hybrid approach: if an event delegate is registered, have a specific thread wait for incoming messages and
use PInvoke to dispatch the event in the hosts own context:
Is it worth having SendMessage be asynchronous? Dunno about that. For simplicity, I would say no. Let the host call
SendMessage from a subthread if needed.
Second, let's examine the "server" point of view. The main difference with a client is that the server will be passively
waiting for connections to arrive. Of course most of the time the host may be both server and client for different tasks.
A host would use the SNE server like this:
The obvious difference with the client is Begin/EndListen and the SetConnectionReceived to indicate the event delegate
that will handle new connections. To keep matters simple, the server opens connections automatically. If the host wants
to refuse the connection, it can close it as soon as the event is received.
The other obvious difference is that sending a message requires a connection id of some sort.
The better alternative to that is to create a connection object that has verbs SendMessage and Close as well as its queue
of received messages. This connection object should be the same that is used in the client side.
20031223
The whole implementation should mostly fit in one class, Alfray.Sne.RSneClient.
From the host point of view there should be some easy methods:
Notes:
20040101
One class doesn't fit all :-)
Here's what I need:
RSneClient should mostly have a static "Connect()" method that will return a RSneConnection.
20031229
One of goals of SNE is to continue playing with TDD and NUnit.
As with anything "new", it's hard to start. Sure I've read
Beck's book,
but when I'm at the point where I want to code, things are not so easy. Blank page syndrome.
20031230
Here I know what I want to code. I'm not sure what to test for. I've seen many comments on weblogs
of testing after coding. I tried that, and it's really boring -- I have the code, it's just for
myself, it may be pretty obvious and either bug free or easy to debug, so why testing? There's no
incentive. The only one that comes to mind is publishing the open source code with tests and say
to the world "see I do have regression/unit testing, ah!". OK that's lame.
The other thing I do not conceptualize well in TDD is the scope of tests. Here I want a
simple communication class. Its public interface will be something like open, send,
receive, close. Let's examine the public method open. I can write a test that
says "open(localhost, some port)". Easy. But that's just one test and inside open will do
a lot of things -- protocol negotiation, sending/receiving data, compressing, etc. How the heck
I am supposed to test all this, it is private inside the class? It's is inside the black box.
Well I suppose I could have the tests be in the class then. That kind of sucks, design wise.
Also all the unit tests I have seen are external classes; most are even external libraries or
in this case external assemblies.
In my case I started with one test class that was inside the main assembly. I didn't like this
approach -- it means the assembly that is shipped embeds all the tests, which does not seem
reasonable. The final assembly should not require a reference on NUnit.Framework to work, nor
does it require the extra binary code of the tests. So I went for a separated assembly containing
all the tests. That doesn't change anything -- how can my test class access private methods of the
class to be tested? It can't. There's no even the notion of "friend classes" like in C++ (and for
many reasons I fear friend classes, they are the goto of oriented-object design imho and as gotos
sometimes they can be useful but most of the time I better do without).
So heck, how can I "drive" the design of my class by testing if the tests won't be able to reach the
methods? Looking at NUnit doesn't solve anything -- they only seem to be using public methods, grrr.
Well I came up with a theoritical solution: subclassing.
So let's try that. More ranting later.
OK 5 mn of testing/coding and I already have some experience:
And I already disagree with one of the principles of TDD: go to the minimum in teensy tiny steps. Sure
that can be cool but in most cases it is not. One of the principles they mention which I don't like is
overdesigning. F.ex. here I want a "bool Connect(string host, int port)" method. The TDD book tells I should
start with "void Connect()", test that first, code it empty, then add one more thing, etc. Sure that's OK but
here I know I will want it. So I might as well put in the return value and the arguments right away and
still test the empty method before doing anything.
The idea of using a derived test class has a limitation: contrary to C++ (I am sure?), C# does not
allow me to call a protected method on another instance than this. I have a class TestA which
derives from A. TestA cannot instantiate A and then call some protected method of the instance of A.
The workaround here is for TestA to create an instance of itself. This is kind of a hack, but it is
pretty safe since TestA mainly consists of unit test methods with few side effects.
20031231
How do I write a test for a synchronous version of RSneListener.Listen?!!
I guess I just don't. Does it make sense to have Listen be synchronous in the first place?
I said to simplify in the first milestone I don't want to use threads on the listen side. That means every
accepted connection will be handled right in place, not forked. The obvious limitation is there can be only
one connection at a time; the obvious benefit is that it is not a headache to debug. How do I write a test
for a call that will never return?
The problem is that I don't really want to test "Listen". What I want to test is the individual elements that
build a connection acceptance... accept, negotiate, etc. If I understand the idea of TDD, that means I do not
start by the "Listen" call. Instead I start by writing an accept and a close; then read something from this
accept; then reply, etc. Use a bottom-up approach here, sort of.
That only confirms my idea that TDD can be better used as a way to enforce tests are written rather than
a way to drive design. I already know the design I want -- it may be wrong or not the most efficient, but at least
I have one in mind. If I have no clue on what to design and how, writing a test is not going to help me any. At the
contrary, if I have no clue and I use TDD to build the design bottom-up, I can see the case where the final design
becomes an unmanageable mess without a global structure. TDD is not the miracle solution to world hunger, contrary
to what its proponents seem to prophetize.
With Nerdkill, I learned that returning boolean values from every function is not a good idea, contrary to what I
was initially believing. My original idea was that most programmer don't check return values, starting with me;
sometimes by lazyness, and many times because I don't even think there's a return value or that I have no error
handling paradigm in mind. Having boolean return values everywhere was stupid -- most of the time a function doesn't need
to return a value (think "remove sprite from list"... there's nothing that can go wrong there that I can test for, unless the
computer is on fire or something -- even if the sprite happens to have already been removed, most of the time it is
just fine and can be ignored, i.e. that is a good reason for a debug.assert, not for returning an error).
So with TDD, I think I see the same logical error: not everything is testable, and not everything needs to be tested.
I'm still not clear where the limit is.
20040104
Must... not... resists... TDD.
OK so last time I had the design nicely evolve. From a single Client class, I introduced (refactored?) the Listen class and
later introduced the Connection class. This last one was merely to be able to put in a common place stuff that would
be in both the client and the listener but I didn't want to test twice. So yes it is a refactor.
Then I hit another wall, actually the same as before: how can I test the listener listening to a connection? I need a
connection to do that, but it is handled asynchronously. And I can't handle it synchronously since I can't have the
same code use Listen and Connect synchronously at the same time... Arghhh. But hey, I found something: ''test-driven
design... that means I should make the design testable''. I have this preconceived idea of the API I want, but it's
the wrong approach. What I have is the preconceived goal of the library and it's top-level public API. That would be my
contract, what the customer in an XP scenario would want. Maybe it is not obvious to me because I'm not used to it,
but I can adapt the socket code so that it be testable.
So how do I test mutually exclusive synchronous code? I don't. Instead I test the individual protocol handling methods
with a text stream that comes, for example, from a String object rather than a live network stream. Typically I would have
a method read a full line of text from the protocol (it's text based mostly after all) then have another method
interpret that text and reply to it. Well that second method can be reading from a string and writing to it. I just need
to use a stream that acts on a string, and AFAIK .Net either has it or I can create it easily. What about that method
that gets the line? I can be tested the same way. Since the protocol is based on receiving and emitting text lines,
a furhter approach would be to manage the input/output as an array of strings and have the tests methods simply compare
to what the protocol is supposed to do.
20040119
Doh! I can't remember what I was doing two weeks ago...
Luckily I leftthat comment saying I wanted to rework my approach, i.e. instead of figuring out
how to test a live connection I should rather test bits of communication using string streams
and then later plug that on the network streams.
Yet I don't really know where to start from.
Normally what I would do in this case is use a top-bottom approach: look at the current structure and start
working on one part (for example sending protocol negotiation data), writing generic functions first and then
refining whilst at the same time the design emerges all by itself (generally implies going back and forth till
it's good too).
Instead here let's try to use the ideas of TDD: let's try the "tiny steps" approach. Something will surely emerge.
The idea here is simply to follow the negotiation workflow. It starts by saying that the client sends
a string, so let's start this way. And instead of sending to a network stream, let's send to a string stream. That also
means that the tested method will receive the stream as argument.
This worked. Redesigned ClientNegotiate as calling a protected method clientNegotiate that acts on a Stream and
testing the protected one using a MemoryStream and in/out String conversions. Then added sendAsciiLine and now in the
process of adding receiveAsciiLine. Not a very fast pace, although that's part of the idea behind (don't go as fast as
you can like a fool, slow down a bit and make sure you actually go in a tested path.)
2. Milestone 1
2.1. Deprecated Project Notice
2.2. Summary
20031223
2.3. Purpose & Goal
20031223
As such, SNE would immediatly classify as a basic peer-to-peer (P2P) protocol.
There is no notion of server vs. client in SNE.
Instead, the core idea for SNE is to be as transparent as possible to the client. Ideally the
client should simply be able to send "something". It occured to be recently that by using the
serialization mechanisms available in .Net or Java I could simply send structs or classes data.
In this context, SNE can be seen as a (very very) naive .Net Remoting clone. This would
of course only work if both ends are using the same platform (.Net or Java on both ends).
Interopability could be achieve by downgrading the exchange to an array of string-formated values
or by using reflexion to list the content of a struct and exchange it with my own serializtion
protocol.
Interopability is not my primary concern here, as I do not expect to use the library in a context
that would require it, but it is nice to keep a backup plan in mind.
Note that the serialization exchange may not be of the most compact format. Initially I'll probably
use the XML Serialization for these platforms. It is easier to debug too.
Similarly local computer IP should not be relied on as it is useless when 2 different NATs talk
together thru the net. This will affect the capacity to do discovery, whenever I want to add
this to the library.
So the ultimate goal here is to use this library in HumCam, and maybe some day in Nodes.
Also it would be my first real project to actually use TDD; I won't do XP stories here, just TDD
based on some initial design specs and see what happens.
2.4. Milestones
20031223
2.5. Milestone 1 - Design
2.5.1. Overview
Ideally the protocol and the classes should be symetric, as there is no server, only two clients of equal level.
In the first version there is no discovery protocol or anything. It is assume that each end knows the IP of the other
end.
library connection at once.
2.5.2. Low-level protocol
2.5.3. Host API
2.6. Implementation
2.7. TDD
The idea is that the test class for X will be derived from class X. That means the tests can
logically access all protected content but none of the private one, which is a reasonable
tradeof. The only drawback is that the class cannot be sealed, but I wasn't planning for that.
The test class can be in the external test assembly, and it can drive the design.
Note that here I can view TestA has some kind of "not so mock" object: the code is not actually testing
A but TestA, yet it's almost like A is being tested (especially as long as no method from TestA overrides
any of A).
Easier said than done :-)
2.8. Plan
20031223 [1.F] VS.Net C# library project with NUnit library
20031223 [1.F] Project on CVS
20031223 [1.F] VS.Net/C# SNE library with separate SNE-NUnit tests library
20031229 [1.F] Created class RSneClient and RSneClientTest
20031229 [2.F] Use TDD. Really. Where do I start today?
20031230 [1.F] Have the test class derive from the tested class (to test protected methods)
20031231 [1.F] Created a class RSneListener and RSneListenerTest
20040101 [2.F] Made the test library into a self-testing application (assembly that runs NUnit tests on itself)
20040101 [1.F] Bug fix: default port for protocol to be 31415 instead of 80 (can test for default port on machine with web server running)
20040101 [1.F] RSneConnectionTest and RSneConnection
20040101 [1.F] RSneClient.Connect, static returns an RSneConnection after opening the TCP port.
20040119 [1.F] RSneConnection: protected clientNegotiate, sendAsciiLine
20040120 [1.N] RSneConnection: receiveAsciiLine
20040120 [2.N] RSneConnection: serverNegotiate
end
This work is licensed by Raphaël Moll under a Creative Commons License.
Color Theme:
Gray
| Blue
| Black | Sand
| Khaki
| Egg
| None
490 accesses, 1 access from 38.107.179.206
Visited 6 times by Google, last 2012/01/10 21:08
Visited 10 times by Yahoo!, last 2011/10/17 13:14
Visited 2 times by MSN, last 2011/06/17 10:43