http://imgur.com/a/4cR1s#0
8 blocks. You have 2 inputs for setting to 0 (bottom) and 1 (top), 1 input for (write->not->read; in the floor) and a 4th input for "mem cell active"
These inputs lead to 4 ANDs (2 right: write, 2 left: read, 2 top: set/get 1, 2 bot: set/get 0), 2 ORs and 2 NOTs in {or -> not2 -> or2 -> not -> or}
Your circuit may look nice, but you need something to access 1 out of many memory cells and switch between read and write mode.
If you use toggle 0->1 and 1->0, you need to check if it is equal to your input first, for the case you want to write not read.
I guess you can't realize this with less blocks that I had. Maybe with less wire-spam? Then I'd like to see your solution.