Longform
Art Math, Math Art - Toward a Boundless Grounded Infinity
Art Math, Math Art - Toward a Boundless Grounded Infinity
John Hanacek
As We May Sketch
Anything digitized has become an abstraction, so let's embrace it. When I draw into a computer with the flourish of my hand we can take it beyond pixels, beyond even vectors, toward universal mapping attempts, toward a truly metamedium.
A curve drawn by hand is a chance to try fitting a function to the line. A function is just waiting to become metaphorical graphics.
Digital ink will move beyond 'networked paper' to become a magical plane where computer vision partners with the human hand functioning as an interactive external imagination. From sketch to code, from code to sketch, no longer a pipeline but rather a constellation of possibilities, an ever-expanding network of opportunities to map expressiveness and flow to logic and math directly.
Example: ChalkTalk
A tantalizing example to ground that fanciful poetry is available open source as webcode called ChalkTalk, which I emailed Ken Perlin about as a grad student before it was open source begging to use because, just wow how cool is this!
ChalkTalk now is open source and can be extended.
Yet this is not the end, rather a door.
What happens in a world with ambient computation and representation capabilities between us to make all this part of daily life? What could we ‘say’ if we could also conjure forth dynamic programs as easily as drawing in mid air?
Use semiotics to create external thinking support, make it interactive as digital inking.
Harness the power of drawing into computers to let a blank canvas become running code, bridge fuzzy expression with rigid logics into an extensible by default play/work-space for all ages.
While in my masters at Georgetown I worked on two papers that attempted to shed light on a latent paradigm for computer interaction centering around digitizer pens and stylus that used drawing as the method for entry at even the lower levels of computer code.
As 3D technologies continue to mature I can see a path of tablets mixed with glasses that open up a kind of [dynamicland] running the full tree version of the little [Chalktalk seed]. We deserve to move beyond drawing dead fish and punching keys for code. I want to dance and sing a runtime, I want to explore our shared mind through a lucid dream as technology, a shared imagination space able to render and instantiate robust computer code.
I love computing's affordances, yet the interface is not yet complete - now it's time to bring the computer to life at the depth of mind with the speed and intuitive action of our hands.
Symbols that Do, Intention that Desires, Hands that Enact
Programming Education for Myself and Visual Thinkers of All Ages
Georgetown Communication, Culture & Technology 2016 Masters Thesis
Computers are absolutely incredible achievements of human ingenuity. Computers are regions of the universe that are under a much tighter level of human control than any other physical arrangement of matter into a technology yet conceived. Inside of microchips, electricity is serving a purpose directly prescribed by a human at a keyboard.
Each computer is a special place in the universe. Computer chips are arrangements of matter that behave so predictably that numbers and mathematical operations can be encoded in patterns of their state. Every active process has a corresponding physical mapping on the microchip expressed as a pattern of semiconductors that are on or off. We see graphics on arrays of pixels, what we do not see are the datastreams coming from immense grids of switches fluttering on and off in programmed informatics. The ability to hold onto huge strings of instructions and do varying operations on those instructions has transformed mathematics into a firehose of computation, so many operations that you can paint entire 3D worlds.
External computation combined with digital networking is the most powerful technology suite humanity has ever created, and already realizing that we are enveloping the world in, to use Luciano Floridi’s term, the infosphere. None of this though is likely news to you. It seems to be common understanding that computers and computer systems are immensely powerful. Much ink and many pixels have been spent discussing the opportunities and pitfalls of an increasingly cybernetic world. Math and programming are understood as extremely influential forces today, with Lawrence Lessig declaring that code is law. With people spending increasingly large amounts of their time in digital platforms code has a more immediate impact on many lives than law now.
If I know something of this, then why is it that I do not really know how to program?
I have been trying. And I’ve been finding joy and frustration. Coming from a graphic design background I am in a foreign place when trying to understand and manipulate programs. I was always fascinated by computers and assembled PCs and hacked around, but I never learned programming. I was math phobic and thus I was programming phobic. That was the narrative I internalized. Only now am I realizing that I can learn math and programming. I can cobble things together, but I do not understand the concepts, I do not feel comfortable, I am always using duct tape, never a nice hand tool. I am on a quest to learn myself and make it easier for everyone else to learn.
I believe that everyone deserves to know how to think systematically and execute their intention in the form of computer programs. The ability to implement formal logic is part of the new literacy.
Dancing Without Music
“Imagine that children were forced to spend an hour a day drawing dance steps on squared paper and had to pass tests in these ‘dance facts’ before they were allowed to dance physically. Would we not expect the world to be full of ‘dancophobes’? Would we say that those who made it to the dance floor and music had the greatest ‘aptitude for dance’?” - Mindstorms (Papert, page 43)
My childhood exposure to math was a series of facts and modules that I needed to repeat exactly for...some reason.
I was raised in a DIY family, with my dad being a competent carpenter and furniture maker. Assembling materials in 3D space is something I have always had a sense for, yet I did not manage to connect the theory of school geometry with practice. I was unable to make the analogy leap from scribbling symbols on paper to genuine internalization of the concept.
So I carried through my life that I was "bad at math" even though I would routinely use logical reasoning to fix broken computers and help my dad with construction and business. I could not see that programming was a different sort of math, I thought I just couldn't do any of it. I became a proficient user of the Adobe suite and settled into an ability to make static 2D designs. I'm finding that my tools were not designed to bridge the gap between graphic design and code. There are systems that allow the creation of interactive mockups, but I have become obsessed with the visceral input that using a digitizer pen to sketech ideas and comment on others' work allows.
In my third semester of CCT I learned how to make images in Processing and found that was the music for me to understand why I would do math, and what math can let me do. I was not unable to do math, I was simply never shown the right connection to spark my understanding.
Now I find myself as someone who is grasping the concepts of programming, but has a huge amount of experience, and an unashamed bias toward, using systems that provide the ability to manipulate on screen elements directly and offer instant, constant feedback. I want to push code around literally.
But why? Why can't I suck it up and learn? I can, and I am, but first we need to understand why I never learned to program in the first place.
Show And Tell
In Don Norman’s “The Design of Everyday Things” he provides ‘principles of good design,’ which are:
Visibility. By looking, the user can tell the state of the device and the alternatives for action.
A good conceptual model. The designer provides a good conceptual model for the user, with consistency in the presentation of operations and results and a coherent, consistent system image.
Good mappings. It is possible to determine the relationships between actions and results, between controls and their effects, and between the system state and what is visible.
Feedback. The user receives full and continuous feedback about the results of actions.
Traditional “typed statement” programming environments live up to none of these criteria. In traditional programming it is often very difficult to determine relationships between operations and results. The feedback is not continuous and because of syntax constraints often a program being modified is not in a state where you can see the results. Systems like spreadsheets are better, since they offer easier to choose alternatives and usually allow you to see the results with just a click, providing a nice feedback loop. Norman concludes that with spreadsheets “it felt as if you were working directly on the problem, not on a computer.”
This is not just for interactive systems. I have embedded Bret Victor's example of a re-designed Social Network Analysis paper, and I urge you to play with it. You will see that the outputs of the desribed algorithms and the picture of the concept is much more apparent when you can just interact with it. We are in the web era now, there is no real reason to subject people to equatiosn alone when we can provide support structures that can help in higher-order comprehension.
I think that it should be much simpler to create the kind of rich interactive media that Victor is demonstrating. We need a graphic design tool for the modern computer era.
I would consider myself a graphic designer so I am well versed in pushing shapes around the screen and now with a tablet computer, drawing curves. In learning about programming and attempting to make my own things I find myself yearning for direct visual manipulation mechanisms. In my primitive state of thinking, I imagine something like Adobe Illustrator if the whole script of the canvas could be accessed. I want to make an object and copy it with a right click. I want to link functions together spatially on a page as I would elements of a graphic design. I want to make mathland a very real idea, instantiate it in computer graphics and give people the ability to get literally hands on with concepts.
I have a few working research questions that are guiding what I am trying to learn in order to actually understand the problem space:
Could I have a development environment that would let me get my ideas into code much faster so that I can focus on trying more complicated logic through the manipulation of visuals that provide constant feedback?
Can we drive this environment with pen input to use interactive sketching to help elucidate the concepts of programming?
Can we create visual metaphors for programming concepts, like objects and dependencies, that users can manipulate in a direct way such as in software with similar affordances of Adobe Illustrator?
Could such an environment allow younger children to organically discover how to realize their imaginations using computers?
What Kinds of Visually-Oriented Programming Tools Are available?
I have used and looked at
which are diagrammatic programming environments. It is really fascinating to manage control flow by connecting “wires” between functions, gates, variables, databases and trigger events. I find them compelling since it presents a level of interactivity that appeals to my mind, and I have even found that these styles of programming work pretty well collaboratively. Two people can look at the diagram at once and both be looking for different bugs or next steps. However, neither of these systems use different icons for different kinds of concepts, so it results in a giant tangled mess that you still have to zoom into in order to actually make sense of.There are modular systems like and but these are geared primarily at children and so come with unnecessary cuteness quotients and limited feature sets. I am not only interested in making games, but want to sketch entire interfaces and databases, and I think kids should be able to head in that direction too if they find themselves moving that way while working.
There is a modern version of Alan Kay's Smalltalk called , which functions as an Integrated Development Environment (IDE). However the interface is based primarily on the WIMP paradigm and has no pen integration. It seems to have a plethora of icons and controls.
There was also a plugin for Adobe Illustrator called that unfortunately broke upon API updates to Illustrator by Adobe. Scriptographer allowed the creation of graphics using javascript and the manipulation of those graphics directly in the Illustrator environment. A designer could create their own mouse-controlled drawing tools, effects that modify existing graphics, and scripts that generate new graphics.
The creator of Scriptographer, Jurg Lehni has since moved the efforts into a javascript library with the help of Jonathan Puckey called . I have been very intrigued by the affordances of paper.js, especially the system that lets users quickly edit and view in the web, similar to and although limited to paper.js implementations.
Examples - The Pen Innovation Awaits
Digitizer pens are critically under-utilized input devices. The systems I have found, along with my personal experience using my Microsoft Surface Pro 4, show a world where using a computer is much more personal and even more powerful. For an entire list go to the pen-based interface section of the bibliography.
There are, in my mind, two main reasons to focus on digitizer pens: expressivity and flow.
With modern digitizer systems the pen is not just a cursor with a binary "click" like a mouse, but also offers pressure levels. When implemented, this allows for finer expression from the user and is a richer datasource for developers than traditional yes/no mouse clicks. Unfortunately, the current digitizer pen systems do not have web APIs, meaning that such experiments in utilizing pressure for interface are bound to locally compiled software.
More importantly though, writing with a pen on paper, a user can experience a state of flow: when their intention is channeled through their hand and into forms of any kind that they need to represent what they desire. Paper has no "tool modes" to select. This is the key to staying in flow: no switching modes. With a pen you are using one tool to express your intention and nuance comes from physical movements of your hands relative to other physical movements that have been recorded as ink. Using a pen on paper satisfies Norman's principles easily.
Paper is obviously limited in its scope though, so the question becomes: can digitzer pen systems be utilized in a similar slow-enducing single mode?
My experience using applications like Adobe Illustrator and Microsoft OneNote is a strange one. Both of these software systems, and indeed most software for desktop computers, were designed with the mouse and mode-switching in mind. So when I want different color inks in OneNote I need to switch modes to another pen. When I want to draw shapes in Illustrator then draw a path I need to be in a specific mode for each one. This ability to have essentially an infinitly-modal pen is very powerful, it is in keeping with the metamedium possibilities of computers and it makes sketching out and implementing graphical ideas much more visceral and intuitive.
Sketches do not need to be bitmaps, they can be comprised of points along vector paths. This allows every point and the delta between the points to already be encoded into a recognition system.
The SILK system is one of the most fascinating, expecially when compared with modern day software design tools, such as Google's and . These systems are set up primarily to create applications in the style that we are now familiar of. They are machinery for creating 'killer apps'. This is helpful if you seek to create apps for consumption, yet it carries a big risk of funneling designers into a trap: if you have a tool that only makes killer apps you are like a carpenter with only a hammer, the world will look a certain way to you and you will probably end up banging away on things when you should have done something else.
That's why SILK is so fascinating, it does not start you in pre-designed templates but instead allows and encourages you to literally draw from your imagination. This creates a tool that is much more flexible and expressive.
What I think we need is a prototyping environment where we can bring in everything that has so far been utilized in interactive computing from the entire WIMP paradigm to multitouch gestures and then set it all free into a free-sketching environment. A combined environment that featured UML diagram recognition, database sketching and interface sketching would allow rapid prototyping of a much larger set of possible interaction paradigms. Does every menu have to go on the upper left side of a mobile app? What would happen if you drew a whole new and crazy kind of interface? Like so many things in design, you often cannot know until you try. I am not arguing that the modern design aesthetic and practices of apps and websites is wrong, I am merely cautioning that we could get stuck in it if our tools are too rigid.
We need a simple prototyping tool that allows us to freely combine different interaction methods based on their affordances. From pen strokes to mouse clicks and finger swipes all as one ‘attention cursor’ for us to dynamically iterate at maximum fidelity with the full capabilities of computer systems. A tool that recognizes diagrams, sketches databases, and creates interfaces would help us prototype many different interaction ideas quickly. Why do all mobile app menus have to be in the upper left corner? Let's try crazy new ideas instead of sticking to the same old design. We shouldn't rely too much on current design trends and tools, or we might get stuck in them.
There is a fine line here between automating design and augmenting human designers. In a fascinating talk by Tim Ewald titled he describes the change from hand tool-based woodworking to today's modern machine woodworking. Of particular min-blowing interest to me is that in the hand tool days blueprints did not have instructions like we know them today, instead they were all based on proportion and geometric rules.
Ewald seems to think he is stretching the analogy of woodworking to programming, but I think it is not being explored enough. I want to experiment with how to literally program with actual hand tools. The visceral artistic connection achieved by working with ones hands could usher in a generation of programmers who are more connected to their work. We could see a shift from pyramids of code to cathedrals of code as
As We May Loop and Blend
Fouconnier and Turner's Conceptual Blending Diagrams
What is human thought anyway? What are we getting up to when we imagine things or reason through problems?
Gilles Fauconnier and Mark Turner attemped to provide a framework for thinking about thinking with their illustration of "conceptual blending."
They illustrate this idea with a riddle and diagram out the steps.
A Buddhist Monk begins at dawn one day walking up a mountain, reaches the top at sunset, meditates at the top for several days until one dawn when he begins to walk back to the foot of the mountain, which he reaches at sunset. Make no assumptions about his starting or stop- ping or about his pace during the trips. Riddle: Is there a place on the path that the monk occupies at the same hour of the day on the two separate journeys?
In Fauconnier and Turner's view, this riddle is solved through the creation of an "integration network" and the manipulation of "cross-space mappings".
The whole sequence that they present is:
Setting up mental spaces
Matching across spaces
Projecting selectively to a blend
Locating shared structures
Projecting backward to inputs
Recruiting new structure to the inputs of the blend
Running various operations on the blend itself
The diagram showing the essential aspects of their theory.
This conception is compelling to me because it is presented immediately with a visual argument. Even if this theory does not completely describe human cognition, it offers a very concrete way to begin building systems that could let users manipulate and blend symbols which stand for "mental spaces" and see if they can get emergent patterns.
"Thanks to a mapping, full-fledged meaning can suddenly appear in a spot where it was entirely unsuspected."
- I Am a Strange Loop (Hofstadter, page 148)
How Would This All Come Together?
Full disclosure I do not know enough about programming to truly articulate the whole dynaPlane interaction paradigm, let along execute it. But I think that a project like this will be the key to me learning programming. What better way to learn math than to try and build a version of mathland?
For me, there are three stages to this dynaPlane vision.
The first stage is allowing programming into the creation of graphical elements in interactive Illustrator-style environments.
The second stage is exploring visual metaphors for programming concepts and experimenting with allowing the manipulation of those metaphorical icons to function as programming itself.
The third stage is exploring machine vision and machine learning to allow the reduction of discrete interaction modes and the development of more conversational, iterative design approaches to creating the desired program, whatever the program seeks to do. This is in an effort to get to no modes, as Larry Tesler urges us to do.
To achieve a version of dynaPlane with no modes would mean that the software system would effectively have to guess what the user is planning on doing next, which is a daunting problem set.
The vision is still very fuzzy in my mind because I have not yet been able to build anything to actually get my hands on. I think this will change. I have just discovered an IDE called which was written in the functional language Clojure. Light Table allows you to string together functions outside of files and inject your own code onto live web pages and many other scripts and markup languages. : instead of functions causing side effects of assigning variables state, the state of the entire program is one value and all objects and variables are referenced, never changed directly only changed inside of functions. At least as far as I've understood. This seems like a powerful conception though, since I can imagine a moment when, if it was in a functional language, an entire windowed operating system could be used as a variable for another function!
All this will surely update as I learn more. I feel as though a weight is being lifted from my shoulders the more I explore the ideas here because even if I can not make these dreams a reality, at least I am showing myself that I can learn to understand programming. I am gaining the literacy of the cybernetic world, and I hope to help pass it on.
Bibliography
Conceptual Foundations
“A Computational Account of Conceptual Blending in Basic Mathematics.” Accessed April 5, 2016. http://ccg.doc.gold.ac.uk/papers/guhe_csr11.pdf.
“A Design Space of Visualization Tasks.” Accessed April 5, 2016. https://pdfs.semanticscholar.org/b768/fd79df84859a93d397e8ac17c346fada0f59.pdf.
Alexander, James C. “Blending in Mathematics.” Semiotica 2011, no. 187 (January 2011): 1–48. doi:10.1515/semi.2011.063.
Alfred North Whitehead & Bertrand Russell. Principia Mathematica Vol 2. Accessed January 19, 2016. http://archive.org/details/PrincipiaMathematicaVol2.
Dourish, Paul, and Victoria Bellotti. “Awareness and Coordination in Shared Workspaces.” In Proceedings of the 1992 ACM Conference on Computer-Supported Cooperative Work, 107–14. CSCW ’92. New York, NY, USA: ACM, 1992. doi:10.1145/143457.143468.
“Existential Graphs MS 514 by Charles Sanders Peirce,” n.d. http://www.jfsowa.com/peirce/ms514.htm.
Grudin, Jonathan. “The Computer Reaches out: The Historical Continuity of Interface Design,” 261–68. ACM Press, 1990. doi:10.1145/97243.97284.
Jeff Gonis. Alan Kay at OOPSLA 1997 - The Computer Revolution Hasnt Happened yet. Accessed May 3, 2016. https://www.youtube.com/watch?v=oKg1hTOQXoY.
John DeNero. Alan Kay — Doing with Images Makes Symbols. Accessed December 15, 2015. https://www.youtube.com/watch?v=kzDpfk8YhlE.
Kay, A., and A. Goldberg. “Personal Dynamic Media.” Computer 10, no. 3 (March 1977): 31–41. doi:10.1109/C-M.1977.217672.
Kay, Alan. “The Center of ‘Why?’” Viewpoints Research Institute, VPRI Memo M-2004-002, 2004. http://www.vpri.org/pdf/m2004002_center.pdf.
———. “‘The Computer Revolution’, ‘Computer Science’, and ‘Software Engineering’ Haven’t Happened Yet.” In Proceedings of the 4th ACM International Conference on Embedded Software, 6–6. EMSOFT ’04. New York, NY, USA: ACM, 2004. doi:10.1145/1017753.1017758.
———. “The Dynabook - Past, Present, and Future.” In Proceedings of the ACM Conference on The History of Personal Workstations, 85 – . HPW ’86. New York, NY, USA: ACM, 1986. doi:10.1145/12178.2533809.
———. “The Power of the Context.” Viewpoints Research Institute, VPRI Memo M-2004-001, 2004. http://www.vpri.org/pdf/m2004001_power.pdf.
Kay, Alan C. “History of Programming Languages, II.” edited by Thomas J. Bergin Jr. and Richard G. Gibson Jr., 511–98. New York, NY, USA: ACM, 1996. http://doi.acm.org/10.1145/234286.1057828.
———. “Microelectronics and the Personal Computer.” Scientific American 237, no. 3 (September 1977): 230–44. doi:10.1038/scientificamerican0977-230.
———. “Smalltalk and Generic Concepts in Programming Languages.” In Proceedings of the International Conference on APL: Part 1, 340 – . APL ’79. New York, NY, USA: ACM, 1979. doi:10.1145/800136.804484.
———. “The Early History of Smalltalk.” In The Second ACM SIGPLAN Conference on History of Programming Languages, 69–95. HOPL-II. New York, NY, USA: ACM, 1993. doi:10.1145/154766.155364. ———. “The Reactive Engine.” Dissertation, University of Utah, 1969.
Lieberman, Henry. Your Wish Is My Command. Morgan Kaufmann, 2001. http://proquestcombo.safaribooksonline.com/9781558606883.
Lorna Uden, Pedro Valderas. “An Activity-Theory-Based Model to Analyse Web Application Requirements.” Text. Accessed March 3, 2016. http://www.informationr.net/ir/13-2/paper340.html.
“On the Cognitive and Logical Role of Image Schemas in Computational Conceptual Blending.” Accessed April 5, 2016. http://ceur-ws.org/Vol-1315/paper9.pdf.
Peirce, Charles S., Christian J. W. Kloesel, Max Harold Fisch, and Peirce Edition Project. Writings of Charles S. Peirce : A Chronological Edition. Volume 8 1890-1892. Bloomington: Indiana University Press, 1982. http://proxy.library.georgetown.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&AuthType=ip,uid&db=e000xna&AN=318243&site=ehost-live&scope=site.
“Peirce’s Logic | Internet Encyclopedia of Philosophy.” Accessed January 19, 2016. http://www.iep.utm.edu/peir-log/.
Rui Oliveira. Bret Victor - Inventing on Principle. Accessed May 3, 2016. https://www.youtube.com/watch?v=PUv66718DII.
“Separating Programming Sheep from Non-Programming Goats.” Accessed May 15, 2016. http://blog.codinghorror.com/separating-programming-sheep-from-non-programming-goats/.
Shin, Sun-Joo, and Eric Hammer. “Peirce’s Deductive Logic.” In The Stanford Encyclopedia of Philosophy, edited by Edward N. Zalta, Fall 2014., 2014. http://plato.stanford.edu/archives/fall2014/entries/peirce-logic/.
“Style as a Choice of Blending Principles.” Accessed April 5, 2016. http://groups.csail.mit.edu/icelab/sites/default/files/pdf/style04.pdf.
“Tatami Homepage.” Accessed April 5, 2016. http://cseweb.ucsd.edu/groups/tatami/.
TheIHMC. Jeannette M. Wing - Computational Thinking and Thinking About Computing. Accessed December 15, 2015. https://www.youtube.com/watch?v=C2Pq4N-iE4I.
“Toward a Computational Theory of Conceptual Metaphor.” Accessed April 5, 2016. https://www.aaai.org/ocs/index.php/FLAIRS/FLAIRS14/paper/viewFile/7857/7887.
“Towards a Cognitive Model of Conceptual Blending.” Accessed April 5, 2016. http://act-r.psy.cmu.edu/wordpress/wp-content/uploads/2012/12/945Guhe.pdf.
Load content from my.spline.design?
Loading external content may reveal information to 3rd parties. Learn more
Load content from www.jhana.zone?
Loading external content may reveal information to 3rd parties. Learn more
Want to print your doc?
This is not the way.
This is not the way.
Try clicking the ⋯ next to your doc name or using a keyboard shortcut (
CtrlP
) instead.