Iterative elements are not going well. Here are notes on this:
31 Dec., 2012
------------------
Worked on the iterative problem. I tried creating a custom foreach Directive for Velocity. First, the Parser will not accept an underscore in the directive name. I had wanted to use the_foreach, but that's no good. Second, the Parser will only allow the argument syntax ( $item in $list ) in the foreach Directive. It's a hard coded literal in the Parser.
So, I tried creating a subclass and/or a wrapper for the foreach Directive. The name can't be anything but foreach. The next effort was a subclass that copied the foreach Directive code entirely and modifying from there. This required creating a ForeachScope wrapper class as well to allow access to the private members of the scope class. The class uses the foreach name and is found before the Foreach class by the Parser. This is fragile. It does allow additional arguments to be passed in. It accepts an optional boolean argument to indicate whether to perform the nesting constructions. All of this could break with changes to Velocity.
The idea was to have a variable called currentLabel and a stack of currentLabels in the Velocimacro library. On the call of a nested foreach, the currentLabel would be pushed on the scope stack. The currentLabel would have the value of the list argument appended to it. As the loop ended, the currentLabel would be replaced by popping the top value off the stack. This allows nested scoping.
Inside the body of the foreach loop, tags that needed to use the currentLabel would call a VelociMacro called nestedBindTo. This would prepend the currentLabel to the property name and place it in the bindTo attribute.
Here's the problem I'm trying to solve. The controller's model has a List of something, say of Object Group. Each Group wants to be a Panel or Accordion or table row. Each Group has a List of Items. These want to be in a table or ordered/unorder list. They want to be able to subscribTo or reRenderOn events, so they need to be bound back to the controller's Model by the JavaScript engine. The end rendered result would look like this:
<ul id="groups" subscribeTo="save" bindTo="groups">
<ul subscribeTo="save" bindTo="groups.0">
<li><input type="text" subscribeTo="save" bindTo="groups.0.item.0" value="Exciting Item One"></li>
<li><input type="text" subscribeTo="save" bindTo="groups.0.item.1" value="Boring Item Two"></li>
</ul>
<ul subscribeTo="save" bindTo="groups.1">
<li><input type="text" subscribeTo="save" bindTo="groups.1.item.0" value="Group 2 Item One"></li>
<li><input type="text" subscribeTo="save" bindTo="groups.1.item.1" value="Group 2 Item Two"></li>
</ul>
</ul>
I could finish my class hack on the foreach Directive, but this is a terrible hack and very fragile. Another option would be to dump the responsibility for nesting on Devlopers. This puts a load of tedious mapping on the Developer and is error prone. I would still provide the currentLabel, scope stack, and nestedBindTo VelociMacro as well as push and popAndSetCurrentLabel methods for the stack. Coding would have to look like this:
#set( $currentLabel = 'groups' )
<ul id="groups" subscribeTo="save" data="#bindTo( 'groups' )">
#foreach( $group in $page.groups )
#push( $currentLabel )
#set( $currentLabel = $currentLabel'.'$foreach.index )
<ul subscribeTo="save" data="#nestedBindTo( $currentLabel )">
#foreach( $item in $group )
#push( $currentLabel )
#set( $currentLabel = $currentLabel'.'$foreach.index )
<li><input type="text" subscribeTo="save" value="#nestedBindTo( $currentLabel )"></li>
#popAndSetCurrentLabel()
#end
#popAndSetCurrentLabel()
#end
This points out a number of flaws in the design. It does not handle non-form elements having a binding where the value gets displayed. As an example, what if the list items above want to have their values displayed as simple text?
<li data="#nestedBindTo( $currentLabel )"> --- How to get and show the value here? How will it be reRendered? --</li>
This wants a completely different solution.
Monday, December 31, 2012
Sunday, December 30, 2012
Something Different - THE Framework
So I find myself with a little time off here at the end of the year. I'd been wanting to try out a few of the many Java web frameworks out there. I'm always looking for "the one". You know, the one that does everything you want effortlessly with no configuration and no glitches. The one that will handle all those outside cases that other frameworks make you really hack the heck out of them to get it work. Yeah, that impossible framework. I checked out the latest Tapestry, tried Wicket, looked at Grails. They all frustrated me fairly quickly. I was looking for immediate solutions to problems I've faced with JSF, facelets, RichFaces, IceFaces, ADF, and Struts. With lots of free time, I started thinking about what that ideal framework would look like.
I started with the usual principles: simplicity and flexibility. I assume that the developer using the framework is like me: intelligent and intelligently lazy. This developer doesn't want to have his hand held or be tied down into one style of solution. This developer also doesn't want to have to think to much about the framework or its configuration. He wants it to allow rapid prototyping that can be quickly adopted as working code.
Oh yeah, and AJAX, lots of AJAX. I mean, really tight AJAX requests and responses. Each submit should send exactly and ONLY what it needs to send. The responses should have just the stuff needed, too. Any tag should be able to register itself as a listener for either or both the send and response to any event.
The events should be able to register themselves, including their name, and what phases they want to use. The phases would be: gather submit data, send a request, and reRender. The event could even skip the AJAX send if it wants to and just be a gather data and reRender. This would be useful especially in development.
The listeners would register themselves just by adding attributes to their tags. To register to be sent to the server, it would add a subscribeTo attribute with the names of the events it wants to be a part of ( of which it wants to be a part, for the grammar nazis ). To register for the reRendering, it would add a reRenderOn attribute with the names of the events of which it wants to be a part.
Now, here's the cool part. In the value attribute for form fields, or the data attribute of ANY tag, it would simply put something like 'value="page.person.address.lineOne"'. If the page defined a model Object for person.address.lineOne with a value like '227 Main St.', the engine would alter that on the server side to be 'value="227 Main St." bindTo="person.address.lineOne"'. This would help the client side JavaScript engine to take the value or data attribute's value and map it to a JavaScript Object graph of person.address.lineOne. The AJAX request would send this as a JSON string to the server. The server would automagically transform this to the page's Object graph.
Once it had processed the request, the server would send an (optional) JSON response. The client side JavaScript engine would receive this and map the response back to the tags that had registered for reRendering.
A templating engine would allow page composition and fragment reuse. Requests for a new page (navigation) would use a POST/Redirect/GET(POST) paradigm.
Above all, the framework would allow maximum flexibility. Provide a means to register and de-register listeners on the fly. It would intercept true submits and turn them into AJAX requests with all values from all bound tags with subscribeTos, but this would be a convenience. Developers would have to include a few key things in their controllers to make things work. A PageMaker would handle the necessary alterations to the templates to add the bindTo attribute.
One sticky part was the amount of raw configuration that would be required. A Maven archetype could set up most of the necessary boilerplate, saving the developer a lot of time. The rest is down to documentation.
Could this really work? I decided to play around. The results aren't fantastic yet, and they have already led to a number of less than ideal solutions. On the whole, it looks pretty cool, and it has been a lot of fun to do. Here's what I came up with.
My name is Thomas Edward Hunt. My monogram, like the kind you get from your aunt on a set of hankies or an ugly sweater, is THE. I decided to create THE framework. It's really more of a stack with some glue code. For the client side, I went with jQuery and jQuery-UI. It has all the selector and AJAX functionality I was looking for, along with all the cool UI stuff. For the templating engine, Velocity seems to have everything required. It allows page composition. It can do the necessary attribute instrumentation ( I think ). I still have a nagging problem with iterative elements. See my notes below for more. For the servlet piece, Jersey has proven very flexible. With JaxB, it can take JSON as an argument and render JSON from an Object graph with little or no effort. Just a little effort gets you the POST/Redirect/Get paradigm. Spring adds the necessary glue for almost everything. Spring Acegi will do the security. Add in the ORM of your choice. I haven't even begun to look at validation or submission value scrubbing/security. One thing at a time.
Here are my working notes to date. These are REALLY rough, but I like to capture my evolving thoughts.
*********************************************************************************
NOTES:
*********************************************************************************
18, Dec., 2012
---------------
Premise:
Build a web application in Jersey and jQuery. The majority of requests will be Ajax calls that will return a JSON object. The HTML will be mostly plain markup. A special attribute on tags will mark what events it will subscribe as a listener for refresh. Whole swaths of HTML may be rewritten in this way while still allowing prototype placeholder text to be put in the HTML. Gotta figure out how to do this.
Jersey will allow composition of HTML by recursively scanning the page templates for composed sub-pages.
On Ajax response, jQuery should allow to find all elements whose subscribe attribute is set to the event at hand.
This wants to be converted to a JSON object. Ajax wants to send a POST of the JSON to Jersey. Jersey will convert that to an expected POJO and process.
Server side tasks:
1. Receive request for page. Get master template. Recurse sub templates and compose page. Parse EL and fill with markup. Blech! (Antlr?)
2. Receive JSON with a request as a POST. Convert JSON and process request. Return JSON or FORWARD to new Resource.
Client Side tasks:
1. build Subscription lists. Probably 2 lists: one for "I'm a part of submits for" and one for "I'm a part of reRenders".
2. Build Ajax framework to get all sendSubscribed elements from an Ajax submit and build JSON object. Submit Ajax request as a POST.
3. Ajax framework to receive all responses. Parse JSON response. Need a way to know where each element of JSON goes in the DOM without encumbering the server with knowing about the page's needs.
4. Intercept full page submits and still perform the mappings needed.
5. Need a way for each page to intercept all requests and responses for pre- and post- processing. Ya just need it.
#####################################################################
FORWARDS IN JERSEY
#####################################################################
Step 1. Get access to HttpServletResponse. To do it declare in your service something like:
@Context
HttpServletResponse _currentResponse;
Step 2. make redirect
...
_currentResponse.sendRedirect(redirect2Url);
EDIT
Well, to call forward method you need get to ServletContext. It is can be resolved the same way as response:
@javax.ws.rs.core.Context
ServletContext _context;
Now _context.forward is available
20 Dec., 2012
------------------
Okay, Here is the stack idea: javascript: jQuery and Bootstrap.js; View: Velocity; Servlet Engine: Jersey ( need to marry with VelocityViewServlet in some way. Custom: event registration for send and reRender, using jQuery. This will need the rewrite in velocity to add attributes of subscribe and reRender as well a tag
that will indicate to what part of the model a tag belongs ( eg, person.address.firstLine ). In other words,
it will need to extend the basic Velocity engine to include adding in the modelRef attribute to point to the
model path ( eg, person.address.firstLine ).
Engine: Jersey: will take in a JSON and put out a JSON ( figure out how to accept and auto-translate JSON
to Java ). Figure out page transitions with Post-Redirect-Get. Figure out how to return Velocity templates.
Wire to Spring, Acegi, Hibernate, etc. and make an archetype. Figure out a Submit intercept to follow regular AJAX course.
Velocity looks like it is extensible. It definitely supports includes, so that' good.
Need to figure out the VelocityViewServlet and Jersey integration. It wants this to be seamless yet extensible. Not sure how to do this. Also, needs a security measure to ensure against cross-site scripting, SQL injection, etc.
26 Dec., 2012
------------------
First, Jersey will take a JSON as input and parse it to the Models it knows.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
GENERAL APPROACH GUIDELINE
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This wants ANY tag to be a part of the conversation with the server. To do this, the general approach will be that the value bound will be the the value attribute if the element involved is an form element such as an input, a select, or a textarea. If the element is not one of these, it should look to a data attribute for the
value.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Now, the tags want two attributes: subscribeTo and reRenderOn. These will have a comma separated list of events in them.
On render of the page, the event handler will somehow magically insert an additional attribute for any element that has one or both of these tags. This extra attribute is the bindTo. It will have the path to the model field such as person.address.lineOne.
This will need to be modified to accomodate iterative things like lists, table rows, dropdown menus, etc. They may look like category.0, category.1, etc.
In the jQuery onload process, there wants to be a method to register all listeners. The idea is to get a list of all elements that have a bindTo attribute. Simple to do with a selector: $('[bindTo]').
Each Ajax event wants to have an event name. This could either be an attribute on the calling element ( button, link, etc ) or a string passed in as a parameter. The latter seems to offer more flexibility. Each event
will have several phases. There should be a means to indicate which phases the event requires. The phases will be
1. Gather data from registered elements with a subscribeTo that contains the event and form an Object from it.
2. Transmit an Ajax request ( possibly with a JSON Object from step 1 ) to the server.
3. Handle reRendering, based on registered elements with the event in their reRenderOn attribute.
Really, it would be cool and very flexible if the event could specify any combination of these three phases. Additionally, the event should be able to pass in an arbitrary Object to use in place of the model. This can be added later.
The call first wants to get a list of all registered elements that have the event name in their subscribeTo attribute. It wants to retrieve the value of each element. The value will be the value attribute if the element is a form element. Otherwise, the value will be in the data attribute. The function will use the element's bindTo in order to add the value to the JSON object that will be passed to the server.
Once the JSON Object has been formed and stringified, a jQuery.post() is made. The callback will handle the response and reRender.
The callback wants to get a list of all registered elements that have the event name in their reRenderOn attribute. It wants to retrieve the value from the JSON Object in the Ajax.data, being mindful of iterative indexes in the bindTo value. It wants to set the new value in the element. It needs to figure out how to make sure that the visible page changes when this value is changed.
Notably, dropdown lists have been problematic.
Runtime gives EventCartridge gives Iterator over ReferenceInsertionEventHandlers gives events I want to mess with ( I think ).
27 Dec., 2012
------------------
Decided to not use Bootstrap.js. It is just not usable at this point. Documentation is lacking and it doesn't
work with IE 7-8. Reality is, a good number of users still use these sub-standard browsers.
Decided to go with jQuery UI.
Odd issue with Accordion under a tab nav. Height of accoridions is zero. Google, please...
StackOverflow tells us: you must initialize the accordion before the tabs!!! Gack! How crude, but it works...
Best practice with jQuery MAY BE ( don't know ) to load all inner widgets before the outer ones.
Got the initial project set up with Spring, Jersey, Velocity, MyBatis. Very primitive but working. Next, need
to add velocity template composition, get sources, and figure out event handling...
29 Dec., 2012
-----------------
Okay, Got jQuery, jQuery-UI, Jersey, Velocity, and Spring set up. Need to bind Velocity into Spring eventually.
Wrote the PageMaker class to handle rendering page templates for all Pages. They just need to extend PageMaker.
May want to change this, so it doesn't consume a Page's only super class... Maybe an injected utility?
It really doesn't want to be a static class.
Dropped the EventHandler solution. The event didn't have enough context. Velocity's parser is an actual language parser from JavaCC. It lexes a stream into tokens, parses the token stream into an AST, and then walks the AST to generate the page from the template and values.
Instead, THE provides a Velocimacro in the library. This macro is available to all templates. Developers will need to use the macro on anything they want mapped to the server's model. For form fields, this will be the value attribute. Any other tag can be bound with the data attribute. For a form field that wants to bind to, for example, HomePage's person.address.lineOne, the developer would put this in the vm file:
<input id="lineOne" type="text" value="#bindTo( 'page.person.address.lineOne' )" />
Note, that the value passed in is a string literal. It has NO Velocity
variable indicator. The macro will turn this into a variable reference and also
use the value for its bindTo value. Likewise, you only need to ever specify
'page' for the page and not the page's actual name. This is handled by the
PageMaker class. The macro actually strips out the "page." part of the string.
That's why PageMaker provides the universal "page" root.
The rendered result would be:
<input id="lineOne" type="text" value="227 Foo St." bindto="person.address.lineOne"/>
30 Dec., 2012
------------------
Moved PageMaker to be an injected class ala Spring. Developers will have to inject this into their pages or just new it up as needed. It has no dependencies as yet, so either way will work. The framework archetype injects it for now in case it adds dependencies later.
Next Steps:
1. Work on page navigation and loading. Looks like Post/Redirect/Get might be the right strategy here.
The client side probably wants to intercept full submits, grab everything registered for any subscribeTo event and push it's current value into the javascript Object that will then be turned into JSON. The default submit will be suppressed, and an AJAX POST will be made instead. The server will receive and process as needed, then send a redirect. This **SHOULD** result in a second GET request for the new page.
The redirect will need to have any needed attributes loaded into the query string.
For later consideration: Should this be a Post/Redirect/POST? Developers could put sensitive stuff in the request...
2. Client Side: Lots of good javascript to write here. Baby steps here.
A. get the registration of event listeners done. This is a list of any elements with a bindTo attribute set with either a subscribeTo or reRenderOn attribute.
NOTE: THE VELOCIMACRO SOLUTION PRESENTS A SERIOUS PROBLEM FOR ITERATIVE ELEMENTS! Sorry to shout, but this needs attention. A separate macro will hopefully solve for this. The whole solution is looking a little shaky.
The development experience is getting a little too involved in the engine. This really wants to be abstracted from the developer.
B. Figure out event registration, including phases required and callbacks. Populate the Object for transmission.
C. Handle the reRender callback.
D. Add methods for registering and un-registering listeners on the fly.
*********************************************************************************
*********************************************************************************
A good portion of the JavaScript in the last TODOs has already been worked out in a proof of concept. I'm now working on getting it to a more robust and true solution. The biggest sticking point is the iterative components. I'm not quite sure whether to figure that out first or just move on the JavaScript solution.
I'd appreciate any feedback. Again, this was just a thought experiment that turned into a bit of code as an exercise. As long as it keeps feeling worth it, I'll keep pursuing it. If someone has already solved a part of this well, let me know. I'd be glad to save a few hours on something that's already solved. I'm already leveraging a huge stack of code to solve well defined problems.
Thanks for reading!
Tom
So I find myself with a little time off here at the end of the year. I'd been wanting to try out a few of the many Java web frameworks out there. I'm always looking for "the one". You know, the one that does everything you want effortlessly with no configuration and no glitches. The one that will handle all those outside cases that other frameworks make you really hack the heck out of them to get it work. Yeah, that impossible framework. I checked out the latest Tapestry, tried Wicket, looked at Grails. They all frustrated me fairly quickly. I was looking for immediate solutions to problems I've faced with JSF, facelets, RichFaces, IceFaces, ADF, and Struts. With lots of free time, I started thinking about what that ideal framework would look like.
I started with the usual principles: simplicity and flexibility. I assume that the developer using the framework is like me: intelligent and intelligently lazy. This developer doesn't want to have his hand held or be tied down into one style of solution. This developer also doesn't want to have to think to much about the framework or its configuration. He wants it to allow rapid prototyping that can be quickly adopted as working code.
Oh yeah, and AJAX, lots of AJAX. I mean, really tight AJAX requests and responses. Each submit should send exactly and ONLY what it needs to send. The responses should have just the stuff needed, too. Any tag should be able to register itself as a listener for either or both the send and response to any event.
The events should be able to register themselves, including their name, and what phases they want to use. The phases would be: gather submit data, send a request, and reRender. The event could even skip the AJAX send if it wants to and just be a gather data and reRender. This would be useful especially in development.
The listeners would register themselves just by adding attributes to their tags. To register to be sent to the server, it would add a subscribeTo attribute with the names of the events it wants to be a part of ( of which it wants to be a part, for the grammar nazis ). To register for the reRendering, it would add a reRenderOn attribute with the names of the events of which it wants to be a part.
Now, here's the cool part. In the value attribute for form fields, or the data attribute of ANY tag, it would simply put something like 'value="page.person.address.lineOne"'. If the page defined a model Object for person.address.lineOne with a value like '227 Main St.', the engine would alter that on the server side to be 'value="227 Main St." bindTo="person.address.lineOne"'. This would help the client side JavaScript engine to take the value or data attribute's value and map it to a JavaScript Object graph of person.address.lineOne. The AJAX request would send this as a JSON string to the server. The server would automagically transform this to the page's Object graph.
Once it had processed the request, the server would send an (optional) JSON response. The client side JavaScript engine would receive this and map the response back to the tags that had registered for reRendering.
A templating engine would allow page composition and fragment reuse. Requests for a new page (navigation) would use a POST/Redirect/GET(POST) paradigm.
Above all, the framework would allow maximum flexibility. Provide a means to register and de-register listeners on the fly. It would intercept true submits and turn them into AJAX requests with all values from all bound tags with subscribeTos, but this would be a convenience. Developers would have to include a few key things in their controllers to make things work. A PageMaker would handle the necessary alterations to the templates to add the bindTo attribute.
One sticky part was the amount of raw configuration that would be required. A Maven archetype could set up most of the necessary boilerplate, saving the developer a lot of time. The rest is down to documentation.
Could this really work? I decided to play around. The results aren't fantastic yet, and they have already led to a number of less than ideal solutions. On the whole, it looks pretty cool, and it has been a lot of fun to do. Here's what I came up with.
My name is Thomas Edward Hunt. My monogram, like the kind you get from your aunt on a set of hankies or an ugly sweater, is THE. I decided to create THE framework. It's really more of a stack with some glue code. For the client side, I went with jQuery and jQuery-UI. It has all the selector and AJAX functionality I was looking for, along with all the cool UI stuff. For the templating engine, Velocity seems to have everything required. It allows page composition. It can do the necessary attribute instrumentation ( I think ). I still have a nagging problem with iterative elements. See my notes below for more. For the servlet piece, Jersey has proven very flexible. With JaxB, it can take JSON as an argument and render JSON from an Object graph with little or no effort. Just a little effort gets you the POST/Redirect/Get paradigm. Spring adds the necessary glue for almost everything. Spring Acegi will do the security. Add in the ORM of your choice. I haven't even begun to look at validation or submission value scrubbing/security. One thing at a time.
Here are my working notes to date. These are REALLY rough, but I like to capture my evolving thoughts.
*********************************************************************************
NOTES:
*********************************************************************************
18, Dec., 2012
---------------
Premise:
Build a web application in Jersey and jQuery. The majority of requests will be Ajax calls that will return a JSON object. The HTML will be mostly plain markup. A special attribute on tags will mark what events it will subscribe as a listener for refresh. Whole swaths of HTML may be rewritten in this way while still allowing prototype placeholder text to be put in the HTML. Gotta figure out how to do this.
Jersey will allow composition of HTML by recursively scanning the page templates for composed sub-pages.
On Ajax response, jQuery should allow to find all elements whose subscribe attribute is set to the event at hand.
This wants to be converted to a JSON object. Ajax wants to send a POST of the JSON to Jersey. Jersey will convert that to an expected POJO and process.
Server side tasks:
1. Receive request for page. Get master template. Recurse sub templates and compose page. Parse EL and fill with markup. Blech! (Antlr?)
2. Receive JSON with a request as a POST. Convert JSON and process request. Return JSON or FORWARD to new Resource.
Client Side tasks:
1. build Subscription lists. Probably 2 lists: one for "I'm a part of submits for" and one for "I'm a part of reRenders".
2. Build Ajax framework to get all sendSubscribed elements from an Ajax submit and build JSON object. Submit Ajax request as a POST.
3. Ajax framework to receive all responses. Parse JSON response. Need a way to know where each element of JSON goes in the DOM without encumbering the server with knowing about the page's needs.
4. Intercept full page submits and still perform the mappings needed.
5. Need a way for each page to intercept all requests and responses for pre- and post- processing. Ya just need it.
#####################################################################
FORWARDS IN JERSEY
#####################################################################
Step 1. Get access to HttpServletResponse. To do it declare in your service something like:
@Context
HttpServletResponse _currentResponse;
Step 2. make redirect
...
_currentResponse.sendRedirect(redirect2Url);
EDIT
Well, to call forward method you need get to ServletContext. It is can be resolved the same way as response:
@javax.ws.rs.core.Context
ServletContext _context;
Now _context.forward is available
20 Dec., 2012
------------------
Okay, Here is the stack idea: javascript: jQuery and Bootstrap.js; View: Velocity; Servlet Engine: Jersey ( need to marry with VelocityViewServlet in some way. Custom: event registration for send and reRender, using jQuery. This will need the rewrite in velocity to add attributes of subscribe and reRender as well a tag
that will indicate to what part of the model a tag belongs ( eg, person.address.firstLine ). In other words,
it will need to extend the basic Velocity engine to include adding in the modelRef attribute to point to the
model path ( eg, person.address.firstLine ).
Engine: Jersey: will take in a JSON and put out a JSON ( figure out how to accept and auto-translate JSON
to Java ). Figure out page transitions with Post-Redirect-Get. Figure out how to return Velocity templates.
Wire to Spring, Acegi, Hibernate, etc. and make an archetype. Figure out a Submit intercept to follow regular AJAX course.
Velocity looks like it is extensible. It definitely supports includes, so that' good.
Need to figure out the VelocityViewServlet and Jersey integration. It wants this to be seamless yet extensible. Not sure how to do this. Also, needs a security measure to ensure against cross-site scripting, SQL injection, etc.
26 Dec., 2012
------------------
First, Jersey will take a JSON as input and parse it to the Models it knows.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
GENERAL APPROACH GUIDELINE
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This wants ANY tag to be a part of the conversation with the server. To do this, the general approach will be that the value bound will be the the value attribute if the element involved is an form element such as an input, a select, or a textarea. If the element is not one of these, it should look to a data attribute for the
value.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Now, the tags want two attributes: subscribeTo and reRenderOn. These will have a comma separated list of events in them.
On render of the page, the event handler will somehow magically insert an additional attribute for any element that has one or both of these tags. This extra attribute is the bindTo. It will have the path to the model field such as person.address.lineOne.
This will need to be modified to accomodate iterative things like lists, table rows, dropdown menus, etc. They may look like category.0, category.1, etc.
In the jQuery onload process, there wants to be a method to register all listeners. The idea is to get a list of all elements that have a bindTo attribute. Simple to do with a selector: $('[bindTo]').
Each Ajax event wants to have an event name. This could either be an attribute on the calling element ( button, link, etc ) or a string passed in as a parameter. The latter seems to offer more flexibility. Each event
will have several phases. There should be a means to indicate which phases the event requires. The phases will be
1. Gather data from registered elements with a subscribeTo that contains the event and form an Object from it.
2. Transmit an Ajax request ( possibly with a JSON Object from step 1 ) to the server.
3. Handle reRendering, based on registered elements with the event in their reRenderOn attribute.
Really, it would be cool and very flexible if the event could specify any combination of these three phases. Additionally, the event should be able to pass in an arbitrary Object to use in place of the model. This can be added later.
The call first wants to get a list of all registered elements that have the event name in their subscribeTo attribute. It wants to retrieve the value of each element. The value will be the value attribute if the element is a form element. Otherwise, the value will be in the data attribute. The function will use the element's bindTo in order to add the value to the JSON object that will be passed to the server.
Once the JSON Object has been formed and stringified, a jQuery.post() is made. The callback will handle the response and reRender.
The callback wants to get a list of all registered elements that have the event name in their reRenderOn attribute. It wants to retrieve the value from the JSON Object in the Ajax.data, being mindful of iterative indexes in the bindTo value. It wants to set the new value in the element. It needs to figure out how to make sure that the visible page changes when this value is changed.
Notably, dropdown lists have been problematic.
Runtime gives EventCartridge gives Iterator over ReferenceInsertionEventHandlers gives events I want to mess with ( I think ).
27 Dec., 2012
------------------
Decided to not use Bootstrap.js. It is just not usable at this point. Documentation is lacking and it doesn't
work with IE 7-8. Reality is, a good number of users still use these sub-standard browsers.
Decided to go with jQuery UI.
Odd issue with Accordion under a tab nav. Height of accoridions is zero. Google, please...
StackOverflow tells us: you must initialize the accordion before the tabs!!! Gack! How crude, but it works...
Best practice with jQuery MAY BE ( don't know ) to load all inner widgets before the outer ones.
Got the initial project set up with Spring, Jersey, Velocity, MyBatis. Very primitive but working. Next, need
to add velocity template composition, get sources, and figure out event handling...
29 Dec., 2012
-----------------
Okay, Got jQuery, jQuery-UI, Jersey, Velocity, and Spring set up. Need to bind Velocity into Spring eventually.
Wrote the PageMaker class to handle rendering page templates for all Pages. They just need to extend PageMaker.
May want to change this, so it doesn't consume a Page's only super class... Maybe an injected utility?
It really doesn't want to be a static class.
Dropped the EventHandler solution. The event didn't have enough context. Velocity's parser is an actual language parser from JavaCC. It lexes a stream into tokens, parses the token stream into an AST, and then walks the AST to generate the page from the template and values.
Instead, THE provides a Velocimacro in the library. This macro is available to all templates. Developers will need to use the macro on anything they want mapped to the server's model. For form fields, this will be the value attribute. Any other tag can be bound with the data attribute. For a form field that wants to bind to, for example, HomePage's person.address.lineOne, the developer would put this in the vm file:
<input id="lineOne" type="text" value="#bindTo( 'page.person.address.lineOne' )" />
Note, that the value passed in is a string literal. It has NO Velocity
variable indicator. The macro will turn this into a variable reference and also
use the value for its bindTo value. Likewise, you only need to ever specify
'page' for the page and not the page's actual name. This is handled by the
PageMaker class. The macro actually strips out the "page." part of the string.
That's why PageMaker provides the universal "page" root.
The rendered result would be:
<input id="lineOne" type="text" value="227 Foo St." bindto="person.address.lineOne"/>
30 Dec., 2012
------------------
Moved PageMaker to be an injected class ala Spring. Developers will have to inject this into their pages or just new it up as needed. It has no dependencies as yet, so either way will work. The framework archetype injects it for now in case it adds dependencies later.
Next Steps:
1. Work on page navigation and loading. Looks like Post/Redirect/Get might be the right strategy here.
The client side probably wants to intercept full submits, grab everything registered for any subscribeTo event and push it's current value into the javascript Object that will then be turned into JSON. The default submit will be suppressed, and an AJAX POST will be made instead. The server will receive and process as needed, then send a redirect. This **SHOULD** result in a second GET request for the new page.
The redirect will need to have any needed attributes loaded into the query string.
For later consideration: Should this be a Post/Redirect/POST? Developers could put sensitive stuff in the request...
2. Client Side: Lots of good javascript to write here. Baby steps here.
A. get the registration of event listeners done. This is a list of any elements with a bindTo attribute set with either a subscribeTo or reRenderOn attribute.
NOTE: THE VELOCIMACRO SOLUTION PRESENTS A SERIOUS PROBLEM FOR ITERATIVE ELEMENTS! Sorry to shout, but this needs attention. A separate macro will hopefully solve for this. The whole solution is looking a little shaky.
The development experience is getting a little too involved in the engine. This really wants to be abstracted from the developer.
B. Figure out event registration, including phases required and callbacks. Populate the Object for transmission.
C. Handle the reRender callback.
D. Add methods for registering and un-registering listeners on the fly.
*********************************************************************************
*********************************************************************************
A good portion of the JavaScript in the last TODOs has already been worked out in a proof of concept. I'm now working on getting it to a more robust and true solution. The biggest sticking point is the iterative components. I'm not quite sure whether to figure that out first or just move on the JavaScript solution.
I'd appreciate any feedback. Again, this was just a thought experiment that turned into a bit of code as an exercise. As long as it keeps feeling worth it, I'll keep pursuing it. If someone has already solved a part of this well, let me know. I'd be glad to save a few hours on something that's already solved. I'm already leveraging a huge stack of code to solve well defined problems.
Thanks for reading!
Tom
Sunday, July 29, 2012
Programming PIC 18F in C
I've been playing around with MPLAB X lately. It's still quite buggy, but I love the convenience of an IDE that manages my code, lets me version the software, program the devices, step debug, etc. It's a shame they chose NetBeans instead of Eclipse. Regardless, some new learnings were in the offing. I wrote some programs in C with Microchips C18 compiler. I learned how to program the chips for In Circuit Debugging (ICD ) with the PicKit2. Wow! It is definitely cool to be able to step debug on the actual circuit.
Not much to show for this first foray. I redid an old routine that reads from the ADC on AN0 and pushes the value onto an LCD screen. The set up and whatnot is the same as posted in earlier blogs. The difference was porting the whole thing to C. The config fuses are set in an include file to get them out of the way. Here it is:
configFuses.h
The source code is fairly compact:
adc_lcd.c
Next up is a hybrid C and assembly program that I've been wanting to do. It reads an analog value from AN0. I took advantage of C to divide the value by 6 and add one. This result is fed to a servo value. The servo is driven by chained interrupts on timers 0 and 1. Timer 0 acts as the controller and interrupts on the period ( 50 Hz or 20ms ). On interrupt, it resets itself and starts timer 1 for the first .65 ms of the duty cycle. It turns on all servos ( in this case only one ). The timer 1 interrupt handles the remaining 1.35 ms of the duty cycle. It sets itself to interrupt every 7.4 us for 180 times. This means that the servos will have a resolution of 180 degrees.
The interrupts are all written in assembly. While they could have been done in C for one servo, this code wants to evolve to control 16 or 17 servos at a time. Each will have an independent value. To control the 18 or more servos for a hexapod, the timings will have to adjust to a PLL driven clock at 48 MHz. That buys 160 instructions per "degree" of resolution to work with ( as opposed to the 40 instructions available at 20 MHz ).
The assembly portion is rammed into the code inline. This led to some interesting knowledge. The program employs a lot of #define statements to make the code more readable and adjustable. All of the delays and interrupt timings are done by define. It should be a simple matter to adjust them to 48 MHz. For the assembly, the pins needed to be defined as well. They could not use the definitions from the chip's include file. As a part of changing the timings for 48 MHz, the assembly will be placed in its own .asm file and linked in. That should simplify things.
The same configFuses.h file was used as above. One thing to note: The MCLR is ON in this file. This allows debugging the program in circuit. It also means a 10k resistor needs to be connected to RE3.
Here is the code for the servo tester as it stands now: adc_servo.c
That's all for now!
Not much to show for this first foray. I redid an old routine that reads from the ADC on AN0 and pushes the value onto an LCD screen. The set up and whatnot is the same as posted in earlier blogs. The difference was porting the whole thing to C. The config fuses are set in an include file to get them out of the way. Here it is:
configFuses.h
The source code is fairly compact:
adc_lcd.c
Next up is a hybrid C and assembly program that I've been wanting to do. It reads an analog value from AN0. I took advantage of C to divide the value by 6 and add one. This result is fed to a servo value. The servo is driven by chained interrupts on timers 0 and 1. Timer 0 acts as the controller and interrupts on the period ( 50 Hz or 20ms ). On interrupt, it resets itself and starts timer 1 for the first .65 ms of the duty cycle. It turns on all servos ( in this case only one ). The timer 1 interrupt handles the remaining 1.35 ms of the duty cycle. It sets itself to interrupt every 7.4 us for 180 times. This means that the servos will have a resolution of 180 degrees.
The interrupts are all written in assembly. While they could have been done in C for one servo, this code wants to evolve to control 16 or 17 servos at a time. Each will have an independent value. To control the 18 or more servos for a hexapod, the timings will have to adjust to a PLL driven clock at 48 MHz. That buys 160 instructions per "degree" of resolution to work with ( as opposed to the 40 instructions available at 20 MHz ).
The assembly portion is rammed into the code inline. This led to some interesting knowledge. The program employs a lot of #define statements to make the code more readable and adjustable. All of the delays and interrupt timings are done by define. It should be a simple matter to adjust them to 48 MHz. For the assembly, the pins needed to be defined as well. They could not use the definitions from the chip's include file. As a part of changing the timings for 48 MHz, the assembly will be placed in its own .asm file and linked in. That should simplify things.
The same configFuses.h file was used as above. One thing to note: The MCLR is ON in this file. This allows debugging the program in circuit. It also means a 10k resistor needs to be connected to RE3.
Here is the code for the servo tester as it stands now: adc_servo.c
That's all for now!
Thursday, May 17, 2012
NTSC Composite Video
As usual, one thing distracts another. The hardware is set up for the first VGA test, but the code has not come out yet. In the meantime, I found an old LCD screen that I wanted to examine. It comes from an old DVD player for the car. The main unit has a DVD player and 7" LCD screen. It also comes with a second 7" LCD that attaches to the main unit via a three prong RCA cable. The signal should be composite video. The screen on the main player is broken, but the second one works. It is small, thin, and portable.
It is ripe for hacking, so VGA will now have to wait. This screen is just waiting for a game of Pong or just a cool, big, text terminal! The NTSC and PAL standards are quite taxing to implement in software. Questions to the Google revealed a few possible shortcuts. Here then are my notes thus far. The project will attempt to write black and white ( and maybe some gray ), using my new Pic18F4550, clocked with PLL at 48MHz. This gives a Pic a working speed of 12 MIPS. The links below provide some of the shortcuts that should make the vertical synching less of a hassle.
==========================
Composite Video
==========================
-------------
17 May, 2012
-------------
References
----------
Rikard Gunee's Pong and Tetris on a Pic
The Post That Gave Me Inspiration
Uzebox
The Shortcuts from AVRFreaks
Going to try the simplest method. This will not try to do interlaced lines. Instead, it will only do the even line field over and over.
Frame = 2 fields of both even lines.
Timings
--------
Line = 63.5 us
262 lines per field
Time per Field = 262 x 63.5 = 16.637 ms
Line Parts
----------
Name Voltage Time Instructions @ 12 Mips
_______________________________________________________
Front Porch .3V 1.4us 17 ( actual 16.8 )
HSynch 0V 4.4us 53 ( actual 52.8 )
Back Porch .3V 4.4us 53 ( actual 52.8 )
Active .3-1.2V 53.3us 639 ( actual 639.6 )
Total Instructions = 762
Freq = 8.333 x 10^-8
Total Time = 762 x 8.333x10^-8 = 63.5us
Vertical Synch
---------------
Try just setting the line to 0V for the whole 63.5us on the 3 VSynch lines.
Field Layout
------------
Lines Function
------------------------------------
3 Dummy H lines
3 VSynch lines
14 Dummy H lines
242 Visible lines
Dummy H line:
Front Porch, HSynch, Back Porch, .3V for active 53.3us
During the active time of the dummy lines, 639 free instructions are available per line.
Use this time to set up the line arrays.
Pin Setup
----------
Pin 1 has a 470 Ohm resistor
Pin 0 has a 1k resistor
Pins Result
---------------
00 0V (synch)
01 .3V (black)
10 .6V (gray)
11 1V (white)
Interrupts
-----------
Use Timer 0 in 16 bit mode. Interrupt every 63.5us ( every 762 instructions at 48MHz ).
65536 - 762 = 64774
In binary = 0b1111110100000110
The High byte is 253
The Low byte is 6
High = 0xFD
Low = 0x06
Need to keep track of line numbers. At start of interrupt, immediately set output to 01 ( black ).
If it is a vertical synch, set output port to 00 ( 0V ) and return to process data.
Else, waste time and do the horizontal synch ( porches and synch ).
If it's a dummy line, return to main for processing.
If it's an active line, process the line.
Need to figure out the line and array housekeeping, too.
Need to look for a way to use the 106 instructions of porch and Hsynch time effectively.
It is ripe for hacking, so VGA will now have to wait. This screen is just waiting for a game of Pong or just a cool, big, text terminal! The NTSC and PAL standards are quite taxing to implement in software. Questions to the Google revealed a few possible shortcuts. Here then are my notes thus far. The project will attempt to write black and white ( and maybe some gray ), using my new Pic18F4550, clocked with PLL at 48MHz. This gives a Pic a working speed of 12 MIPS. The links below provide some of the shortcuts that should make the vertical synching less of a hassle.
==========================
Composite Video
==========================
-------------
17 May, 2012
-------------
References
----------
Rikard Gunee's Pong and Tetris on a Pic
The Post That Gave Me Inspiration
Uzebox
The Shortcuts from AVRFreaks
Going to try the simplest method. This will not try to do interlaced lines. Instead, it will only do the even line field over and over.
Frame = 2 fields of both even lines.
Timings
--------
Line = 63.5 us
262 lines per field
Time per Field = 262 x 63.5 = 16.637 ms
Line Parts
----------
Name Voltage Time Instructions @ 12 Mips
_______________________________________________________
Front Porch .3V 1.4us 17 ( actual 16.8 )
HSynch 0V 4.4us 53 ( actual 52.8 )
Back Porch .3V 4.4us 53 ( actual 52.8 )
Active .3-1.2V 53.3us 639 ( actual 639.6 )
Total Instructions = 762
Freq = 8.333 x 10^-8
Total Time = 762 x 8.333x10^-8 = 63.5us
Vertical Synch
---------------
Try just setting the line to 0V for the whole 63.5us on the 3 VSynch lines.
Field Layout
------------
Lines Function
------------------------------------
3 Dummy H lines
3 VSynch lines
14 Dummy H lines
242 Visible lines
Dummy H line:
Front Porch, HSynch, Back Porch, .3V for active 53.3us
During the active time of the dummy lines, 639 free instructions are available per line.
Use this time to set up the line arrays.
Pin Setup
----------
Pin 1 has a 470 Ohm resistor
Pin 0 has a 1k resistor
Pins Result
---------------
00 0V (synch)
01 .3V (black)
10 .6V (gray)
11 1V (white)
Interrupts
-----------
Use Timer 0 in 16 bit mode. Interrupt every 63.5us ( every 762 instructions at 48MHz ).
65536 - 762 = 64774
In binary = 0b1111110100000110
The High byte is 253
The Low byte is 6
High = 0xFD
Low = 0x06
Need to keep track of line numbers. At start of interrupt, immediately set output to 01 ( black ).
If it is a vertical synch, set output port to 00 ( 0V ) and return to process data.
Else, waste time and do the horizontal synch ( porches and synch ).
If it's a dummy line, return to main for processing.
If it's an active line, process the line.
Need to figure out the line and array housekeeping, too.
Need to look for a way to use the 106 instructions of porch and Hsynch time effectively.
Sunday, May 6, 2012
VGA Wiring, Fast 16 Bit Counters
Here are a few brief notes/findings.
Cheap Wire
Home Depot in the States sells Cat 5e riser network cable. This is 24 AWG solid core wire. There are 8 wires in the cable. These work fairly well as jumper wires for breadboard. Left in the insulator, they make really good connection wires. There are eight of them, so that's a parallel byte or a full register. In the case of the line follower I want, that's the six input lines and the power and ground. One short of what is needed ( the LED on/off line ). Here's the cool part: THE PRICE! A 20 foot roll of a single wire at 22 AWG at Radio Shack is a bit over $8.00 USD. The Cat 5e Riser ( not Plenum ) cable is $0.18 per foot. And that's for eight wires. The stuff can be used like Kleenex at that price. Plenum cable is not much more expensive at about $0.59 per foot. This stuff is going on almost everything project from now on.
VGA
A VGA driver is the next experiment on the docket. Lucid Science has a great tutorial. It's written for an Atmel AVR in assembly. With a datasheet and maybe a tutorial, it's pretty easy to adapt to a PIC. The main problem with the PIC is speed. Even with an 18F microcontroller running at 96 MHz with PLL, the processor gets 48 MHz from that. Each instruction takes four clocks, so the effective MIPS is 12. This is slow and it makes conversion a bit tricky. It should be possible to get a test display of Lucid Science's RRGGBBII palette to work. Even with SRAM, the operable resolution for graphics will be pretty low, though. Still, it should be fun to try.
To that end, I wired up a VGA female cable, using some newly purchased Cat 5e cable. As a reminder to myself:
Fast 16 Bit Counter
Now for some cool news. The 18F series has made some big improvements. First, you can avoid the Read-Modify-Write problem by writing directly to the Latch registers for the ports. Next, they include a new SFR, WREG. This shadows the W register. Unlike the W register, it allows direct math and manipulation. Mathematical operations on the WREG can be interchanged with W reserved operations like movlw and addwf.
Tonight proved two other operations unknown to me. First, it is possible to maintain a 16 Bit increment ( an increment over 2 bytes ) in only 2 instructions. The W register must be zero for this to work. The lower byte is incremented as usual with incf:
movlw 0x000
incf myRegLow,F
Then, the magical addwfc is invoked. This adds the W register ( value zero ) to the higher byte as well as the Carry bit from the last operation ( the incf ). This effectively increments the higher byte only when the lower byte overflows to 256 ( ie, back to zero ).
addwfc myRegHigh,F
Wait, wait. Don't order yet! Now how much would you pay? If you act now, we'll throw in the fact that you can do mathematical operations directly on the Latch registers! This means that if the SRAM address for VGA output needs to be incremented very quickly through 16 bits worth of addresses ( and it does ), it is possible. If LATB holds the lower address byte and LATD holds the higher byte:
movlw 0x000
beginAddressLoop:
incf LATB,F
addwfc LATD,F
bra beginAddressLoop
This, of course, is nothing like what will be used. It does illustrate the point. These little discoveries will drastically reduce the cycles needed for the various operations needed for VGA output.
Cheap Wire
Home Depot in the States sells Cat 5e riser network cable. This is 24 AWG solid core wire. There are 8 wires in the cable. These work fairly well as jumper wires for breadboard. Left in the insulator, they make really good connection wires. There are eight of them, so that's a parallel byte or a full register. In the case of the line follower I want, that's the six input lines and the power and ground. One short of what is needed ( the LED on/off line ). Here's the cool part: THE PRICE! A 20 foot roll of a single wire at 22 AWG at Radio Shack is a bit over $8.00 USD. The Cat 5e Riser ( not Plenum ) cable is $0.18 per foot. And that's for eight wires. The stuff can be used like Kleenex at that price. Plenum cable is not much more expensive at about $0.59 per foot. This stuff is going on almost everything project from now on.
VGA
A VGA driver is the next experiment on the docket. Lucid Science has a great tutorial. It's written for an Atmel AVR in assembly. With a datasheet and maybe a tutorial, it's pretty easy to adapt to a PIC. The main problem with the PIC is speed. Even with an 18F microcontroller running at 96 MHz with PLL, the processor gets 48 MHz from that. Each instruction takes four clocks, so the effective MIPS is 12. This is slow and it makes conversion a bit tricky. It should be possible to get a test display of Lucid Science's RRGGBBII palette to work. Even with SRAM, the operable resolution for graphics will be pretty low, though. Still, it should be fun to try.
To that end, I wired up a VGA female cable, using some newly purchased Cat 5e cable. As a reminder to myself:
| Pin | Wire Color(s) | Value |
|---|---|---|
| 1 | Orange | Red |
| 2 | Green | Green |
| 3 | Blue | Blue |
| 5,6,7,8,10 | White/Green White/Blue White/Orange |
Ground |
| 13 | Brown | Hz Synch |
| 14 | White/Brown | Vt Synch |
Fast 16 Bit Counter
Now for some cool news. The 18F series has made some big improvements. First, you can avoid the Read-Modify-Write problem by writing directly to the Latch registers for the ports. Next, they include a new SFR, WREG. This shadows the W register. Unlike the W register, it allows direct math and manipulation. Mathematical operations on the WREG can be interchanged with W reserved operations like movlw and addwf.
Tonight proved two other operations unknown to me. First, it is possible to maintain a 16 Bit increment ( an increment over 2 bytes ) in only 2 instructions. The W register must be zero for this to work. The lower byte is incremented as usual with incf:
movlw 0x000
incf myRegLow,F
Then, the magical addwfc is invoked. This adds the W register ( value zero ) to the higher byte as well as the Carry bit from the last operation ( the incf ). This effectively increments the higher byte only when the lower byte overflows to 256 ( ie, back to zero ).
addwfc myRegHigh,F
Wait, wait. Don't order yet! Now how much would you pay? If you act now, we'll throw in the fact that you can do mathematical operations directly on the Latch registers! This means that if the SRAM address for VGA output needs to be incremented very quickly through 16 bits worth of addresses ( and it does ), it is possible. If LATB holds the lower address byte and LATD holds the higher byte:
movlw 0x000
beginAddressLoop:
incf LATB,F
addwfc LATD,F
bra beginAddressLoop
This, of course, is nothing like what will be used. It does illustrate the point. These little discoveries will drastically reduce the cycles needed for the various operations needed for VGA output.
Friday, May 4, 2012
Hand Made Board - Finished!
The Dremel drill bits from Amazon arrived. They come in a packet of 4 and are 1/32". Man, are they small! They fit in the Dremel chuck nicely, but it took a bit of fiddling to get them in straight. The problem with the Dremel is that the hold button needs to be depressed as you tighten. The whole operation would be a lot easier with three hands. I used my trusty Dremel drill press and banged out the holes. It was very easy. I took my time. The copper around the holes actually nudge the bit where it needs to go. The Dremel was set to 15,000 RPMs. The bit dwells a bit at the top of the cut, then a smooth plunge finishes off each hole. Here's the result:
Soldering was very different from either a proto-board or a professional PCB. The solder and heat want to go everywhere. The ground plane was particularly a problem. I have to admit, I rushed the soldering. I was so excited that everything looked good. I just wanted to see the board done. Here's the top with the pins and sockets added:
Here's a picture of the bottom solder that's as bad as the soldering job:
I printed out a pin "cheat sheet" for the microcontroller in lieu of a stencil:
Now, to test it out with a hello world style blinking LED!
Soldering was very different from either a proto-board or a professional PCB. The solder and heat want to go everywhere. The ground plane was particularly a problem. I have to admit, I rushed the soldering. I was so excited that everything looked good. I just wanted to see the board done. Here's the top with the pins and sockets added:
Here's a picture of the bottom solder that's as bad as the soldering job:
I printed out a pin "cheat sheet" for the microcontroller in lieu of a stencil:
Now, to test it out with a hello world style blinking LED!
Sunday, April 29, 2012
Etching Your Own PCBs
I decided to have a go at etching my own PCB. It looks like it will be a while before I get around to building my CNC router, so I thought I'd give a thought to making boards the chemical way. I've been a bit put off by the idea of having strong acids in the house, so I never got around to trying this. As luck would have it, I've been following Quinn Dunki's "Blondie Hacks" blog for a bit, watching her develop her own computer from scratch. She posted a blog, detailing her PCB making process. I was surprised to find that she does her etching with Hydrogen Peroxide, White Vinegar, and Salt. That sounded a lot safer than HCL! I checked with my wife ( a chemistry ( and physics, and calculus ) teacher ). She gave me the basics of the chemistry and seemed to think it would work. Here's what happened.
First, I read Quinn's blog about a hundred times. She links to the Sparkfun tutorials about EagleSoft CAD. These are the same tutorials I had used to get started. They are excellent, and Eagle is free ( as in beer ). She also has links to most of the materials you will need. I got alot of stuff from the dollar store ( peroxide, acetone, tupperware, foam brushes, picture frame, cotton balls, funnel, etc ). I was ordering from mouser.com anyway, so I got my presensitized PCB and Positive Developer there. I can't say enough good things about mouser. Once you get used to their search filters, you can find almost anything there.
I have to take a detour here and talk about mouser a bit more. I thought their shipping prices were a bit high. Then, I got my package. I had ordered some new microcontrollers, some SOJ SRAM, some 74HC logic chips, and other stuff with my PCB and developer. Wow! I thought they were delivering medical supplies. The careful packaging, the anti-static bubble wrap, all kind of blew me away. The craziest was the packaging on the SRAM. It came in an anti-static silver baggy, of course. On that was the usual yellow warning label that this was a static sensitive device. In addition was a label that looked like the instructions for handling an human organ tranfer. We're talking about expiration times, humidity controls, temperature controls; you name it. Inside the bag ( once I got the courage to open it ), I found 3 silica gel packs and a Ph sensor card along with a small black box. The box had additional warning and control information. Inside the box, in tons of carefully arranged anti-static foam, were my two little SRAMs. Did I mention that everything was in English? And most of the entries were hand written. I was very impressed.
Anyway, back to board etching. I decided to start with a simple board. I got a new Pic18F4550 and decided to make a carrier board for it. First up was the schematic in Eagle:
From this, I made a board, following Quinn's advice on hole size, ground plane, etc. I won't repeat everything she says. Go read her blog. It's well done. Here's the board:
Nice and simple, and it won't need any jumpers. I did forget to add in the decoupling cap though. Doh!
From there, I printed the board as a PDF file and edited it with GIMP. I printed this board along with the next one I'll try ( line follower sensor ) on acetate. I found some HP transparencies that work with an inkjet printer. They have a rough texture on the printing side that helps absorb and hold the ink. They dry quickly and come out very dark:
Still, I doubled them up as Quinn suggests. This seems to have made a difference. Taping them together wasn't as much of a pain as I thought it would be:
Then, it was on to the darkroom. I used my basement workshop/office. Luckily, I have a ton of red LEDs sitting around down there. I threw ten on my breadboard to use as a darkroom light:
It was dark, but it did the job. I had a bit of trouble when I put the board onto the glass. It was tough to see if the board was aligned with the artwork. For no particular reason, here are the LEDs on:
Behind the LEDs, you can see the bench power supply I made from an old PC PSU. I forgot to blog that one up as I did it. Oh, well. It's great. Right now, It supplies 12, 5, and 3.3V with plenty of amps. Here's the board under exposure:
Here's the developer bath all ready to go. I didn't have time or light to take pictures here.
Behind the tupperware, you can see my three new stepper motors. I hope to use them to build my CNC. They're going to be too slow, but they should work. Also, to the left, you can see the proto board for the line sensor and my robot behind it. To the right, you can see a bunch of new chips I got from Mouser.
After developing and a quick rinse, it was on to the garage for etching. I mixed equal parts of hydrogen peroxide and vinegar. The etching was *really* slow. I'm glad I brought along a healthy cup of salt. The salt acts as a catalyst for the reaction. I think my tupperware is too small. Next time, I'll use a bigger one to speed up the reaction. Here's the board just dropped in the etchant:
And here is the final product:
I left the resist on for now. I'm waiting for my drill bits to arrive. With luck, I won't peel off too many pads!
First, I read Quinn's blog about a hundred times. She links to the Sparkfun tutorials about EagleSoft CAD. These are the same tutorials I had used to get started. They are excellent, and Eagle is free ( as in beer ). She also has links to most of the materials you will need. I got alot of stuff from the dollar store ( peroxide, acetone, tupperware, foam brushes, picture frame, cotton balls, funnel, etc ). I was ordering from mouser.com anyway, so I got my presensitized PCB and Positive Developer there. I can't say enough good things about mouser. Once you get used to their search filters, you can find almost anything there.
I have to take a detour here and talk about mouser a bit more. I thought their shipping prices were a bit high. Then, I got my package. I had ordered some new microcontrollers, some SOJ SRAM, some 74HC logic chips, and other stuff with my PCB and developer. Wow! I thought they were delivering medical supplies. The careful packaging, the anti-static bubble wrap, all kind of blew me away. The craziest was the packaging on the SRAM. It came in an anti-static silver baggy, of course. On that was the usual yellow warning label that this was a static sensitive device. In addition was a label that looked like the instructions for handling an human organ tranfer. We're talking about expiration times, humidity controls, temperature controls; you name it. Inside the bag ( once I got the courage to open it ), I found 3 silica gel packs and a Ph sensor card along with a small black box. The box had additional warning and control information. Inside the box, in tons of carefully arranged anti-static foam, were my two little SRAMs. Did I mention that everything was in English? And most of the entries were hand written. I was very impressed.
Anyway, back to board etching. I decided to start with a simple board. I got a new Pic18F4550 and decided to make a carrier board for it. First up was the schematic in Eagle:
From this, I made a board, following Quinn's advice on hole size, ground plane, etc. I won't repeat everything she says. Go read her blog. It's well done. Here's the board:
Nice and simple, and it won't need any jumpers. I did forget to add in the decoupling cap though. Doh!
From there, I printed the board as a PDF file and edited it with GIMP. I printed this board along with the next one I'll try ( line follower sensor ) on acetate. I found some HP transparencies that work with an inkjet printer. They have a rough texture on the printing side that helps absorb and hold the ink. They dry quickly and come out very dark:
Behind the LEDs, you can see the bench power supply I made from an old PC PSU. I forgot to blog that one up as I did it. Oh, well. It's great. Right now, It supplies 12, 5, and 3.3V with plenty of amps. Here's the board under exposure:
Here's the developer bath all ready to go. I didn't have time or light to take pictures here.
After developing and a quick rinse, it was on to the garage for etching. I mixed equal parts of hydrogen peroxide and vinegar. The etching was *really* slow. I'm glad I brought along a healthy cup of salt. The salt acts as a catalyst for the reaction. I think my tupperware is too small. Next time, I'll use a bigger one to speed up the reaction. Here's the board just dropped in the etchant:
Monday, March 26, 2012
64-bit Woes, Pickit 2, and pk2cmd
Okay, so the old laptop finally died. I'd been calling it the "ghetto laptop" for a few months. Power issues, overheating, cracked the hinge on the lid, etc. The cracked lid gave rise to the nickname ghetto laptop. I had to open and close it by placing a flathead screwdriver in the security lock hole. All it needed was some duct tape on the cover! Then the DVD drive died. Oh well. Then the screen went out.
In panic, I ordered the third most expensive laptop I could find without doing any real research. It's a great machine... unless you like Linux. I do. The problem is that it has an Optimus setup. This means that it has an integrated Intel graphics card hooked directly to the monitor. It also has a dedicated Nvidia graphics card with dedicated 1.5 GB of Ram. This card is hooked *through* the Intel card. The software is meant to handle the switching on/off of the 2 cards. No one felt the need to provide these specs to the open source community, so Linux support is primitive at best at present.
I managed to get everything setup, and I'm now running bumblebee. Great work and thanks to the folks who are working on the bumblebee team! This allows me to run normal stuff from the Intel card and start apps that need better graphics with the Nvidia card.
I also decided that 64-bit OSes were common enough now that I would install one, too. Turns out that this caused a few headaches. One of the worst was with my Pickit 2 programmer. The open source programming software, pk2cmd, would only work with my 16F88 chips! I posted to the pickit-devel group on Google Groups. I managed to attract the attention of two of the main developers of the software, Jeff Post and Xiaofan Chen. After much testing, I ended up compiling pk2cmd as a 32 bit app. It looks like some work would have to be done to make it work for my hardware and kernel under 64 bit. As the programmer and software is pretty old, I doubt that much will (or should) be done.
As notes to myself ( and anyone who may need this ), here's how I got it to compile as a 32 bit app under a 64 bit system. As I don't really do C programming, this was more of a challenge than I expected. The end solution is quite simple.
I pulled in the libusb-devel packages, g++, etc. as recommended in the readme. I also needed to pull in the libusb-0.1.so.4.4.4 library for 32 bit. I altered the Makefile for 32 bit compilation by adding the switch '-m32' ( without the quotes ) to the definition of OPTS:
OPTS=-Wall -m32 -D_GNU_SOURCE $(DBG)
With this, make linux was able to compile the program, but it could not find the library libusb. So, I su-ed to root and found the 32-bit libusb-0.1.so.4.4.4 library in /lib. I changed directories to /usr/lib and made a symbolic link to it as follows:
ln -s /lib/libusb-0.1.so.4.4.4 libusb.so
I then changed the LDFLAGS definition in the Makefile to look like this:
LDFLAGS=-L/usr/lib -lusb
With this, make linux was able to link the files and create the executable. Su and make install had everything working the way it used to. Huzzah!
In panic, I ordered the third most expensive laptop I could find without doing any real research. It's a great machine... unless you like Linux. I do. The problem is that it has an Optimus setup. This means that it has an integrated Intel graphics card hooked directly to the monitor. It also has a dedicated Nvidia graphics card with dedicated 1.5 GB of Ram. This card is hooked *through* the Intel card. The software is meant to handle the switching on/off of the 2 cards. No one felt the need to provide these specs to the open source community, so Linux support is primitive at best at present.
I managed to get everything setup, and I'm now running bumblebee. Great work and thanks to the folks who are working on the bumblebee team! This allows me to run normal stuff from the Intel card and start apps that need better graphics with the Nvidia card.
I also decided that 64-bit OSes were common enough now that I would install one, too. Turns out that this caused a few headaches. One of the worst was with my Pickit 2 programmer. The open source programming software, pk2cmd, would only work with my 16F88 chips! I posted to the pickit-devel group on Google Groups. I managed to attract the attention of two of the main developers of the software, Jeff Post and Xiaofan Chen. After much testing, I ended up compiling pk2cmd as a 32 bit app. It looks like some work would have to be done to make it work for my hardware and kernel under 64 bit. As the programmer and software is pretty old, I doubt that much will (or should) be done.
As notes to myself ( and anyone who may need this ), here's how I got it to compile as a 32 bit app under a 64 bit system. As I don't really do C programming, this was more of a challenge than I expected. The end solution is quite simple.
I pulled in the libusb-devel packages, g++, etc. as recommended in the readme. I also needed to pull in the libusb-0.1.so.4.4.4 library for 32 bit. I altered the Makefile for 32 bit compilation by adding the switch '-m32' ( without the quotes ) to the definition of OPTS:
OPTS=-Wall -m32 -D_GNU_SOURCE $(DBG)
With this, make linux was able to compile the program, but it could not find the library libusb. So, I su-ed to root and found the 32-bit libusb-0.1.so.4.4.4 library in /lib. I changed directories to /usr/lib and made a symbolic link to it as follows:
ln -s /lib/libusb-0.1.so.4.4.4 libusb.so
I then changed the LDFLAGS definition in the Makefile to look like this:
LDFLAGS=-L/usr/lib -lusb
With this, make linux was able to link the files and create the executable. Su and make install had everything working the way it used to. Huzzah!
Monday, January 16, 2012
IR Distance Sensing
Finally got back into distance sensing today. Had the day off for MLK, jr. Did a lot more research and RTFMing. Hooked up a test rig with the following notes:
Distance Detection with IR Led and IR Phototransistor
================================
2012.01.16 -
-----------------
So, it's back to distance detection with IR Leds and IR phototransistors.
I used the IR pair from Radio Shack. A few comments on these, so I don't
forget yet again, I'm sick of re-learning this. The dark one is the emitter.
The clear one is the detector. The emitter has the following key properties:
Forward Current: 150 mA
Forward Voltage: 1.3 V typical. max is 1.7 V
Reverse Voltage: 5 V
Wavelength: 950 nm
Resistance is calculated as always for a diode:
Vs = voltage source
Vf = forward voltage
I = forward current
So,
Vs - Vf
_______ = resistance for max current.
I
For the Radio Shack emitter at 5V source:
5 - 1.3 / .150 = 24.66666
I use a 33 Ohm resistor to get ~112mA.
The big reminder:
1. The long lead does not always mean hook to Positive!!!!
On the emitter, the long lead corresponds to the flat part of the casing.
This corresponds to the Anode ( positive ) lead of the Led.
On the phototransistor, the long lead hooks to the flat part of the part.
The flat part, short lead is the Collector and goes to positive!!!
The round part, long lead goes to the emitter ( ground ).
o Control the IR Led from the microcontroller by triggering an NPN transistor.
I got good results with a 2N3904. I used a 1k resistor to the base.
o The emitter uses a 33 Ohm resistor as calculated above. If you can find
something closer to 25 Ohms, use that.
o The detector gets unresisted 5V to the collector. The emitter goes to ground
via a 100k resistor. The detection line to the microcontroller is hooked
to the emitter.
Results:
----------
Using a hand or white piece of paper, good results went almost 30cm.
Non-precise measurements:
value | value
cm | hand | white paper
--------------------------------
1 | 553 | 1023
2 | 391 | 820
3 | 279 | 620
4 | 199 | 438
5 | 162 | 351
6 | 126 | 272
7 | 113 | 231
8 | 98 | 194
9 | 89 | 166
10 | 79 | 144
11 | 75 | 132
12 | 70 | 118
13 | 67 | 109
14 | 63 | 99
15 | 61 | 88
16 | 57 | 83
18 | 52 | 74
20 | 50 | 66
22 | 48 | 61
24 | 46 | 56
26 | 45 | 54
28 | 44 | 51
30 | 44 | 49
I hooked up a test rig to get these results. Here is the code:
sense source code
As usual, this is assembly source code for a PIC microcontroller from Microchip. In this case, I'm using my little work horse, the 16F88, running at 8 MHz.
Here are some photos of the rig on breadboard:
I'll whip up a schematic soon.
EDIT:
And here it is in somewhat rough shape:
Distance Detection with IR Led and IR Phototransistor
================================
2012.01.16 -
-----------------
So, it's back to distance detection with IR Leds and IR phototransistors.
I used the IR pair from Radio Shack. A few comments on these, so I don't
forget yet again, I'm sick of re-learning this. The dark one is the emitter.
The clear one is the detector. The emitter has the following key properties:
Forward Current: 150 mA
Forward Voltage: 1.3 V typical. max is 1.7 V
Reverse Voltage: 5 V
Wavelength: 950 nm
Resistance is calculated as always for a diode:
Vs = voltage source
Vf = forward voltage
I = forward current
So,
Vs - Vf
_______ = resistance for max current.
I
For the Radio Shack emitter at 5V source:
5 - 1.3 / .150 = 24.66666
I use a 33 Ohm resistor to get ~112mA.
The big reminder:
1. The long lead does not always mean hook to Positive!!!!
On the emitter, the long lead corresponds to the flat part of the casing.
This corresponds to the Anode ( positive ) lead of the Led.
On the phototransistor, the long lead hooks to the flat part of the part.
The flat part, short lead is the Collector and goes to positive!!!
The round part, long lead goes to the emitter ( ground ).
o Control the IR Led from the microcontroller by triggering an NPN transistor.
I got good results with a 2N3904. I used a 1k resistor to the base.
o The emitter uses a 33 Ohm resistor as calculated above. If you can find
something closer to 25 Ohms, use that.
o The detector gets unresisted 5V to the collector. The emitter goes to ground
via a 100k resistor. The detection line to the microcontroller is hooked
to the emitter.
Results:
----------
Using a hand or white piece of paper, good results went almost 30cm.
Non-precise measurements:
value | value
cm | hand | white paper
--------------------------------
1 | 553 | 1023
2 | 391 | 820
3 | 279 | 620
4 | 199 | 438
5 | 162 | 351
6 | 126 | 272
7 | 113 | 231
8 | 98 | 194
9 | 89 | 166
10 | 79 | 144
11 | 75 | 132
12 | 70 | 118
13 | 67 | 109
14 | 63 | 99
15 | 61 | 88
16 | 57 | 83
18 | 52 | 74
20 | 50 | 66
22 | 48 | 61
24 | 46 | 56
26 | 45 | 54
28 | 44 | 51
30 | 44 | 49
I hooked up a test rig to get these results. Here is the code:
sense source code
As usual, this is assembly source code for a PIC microcontroller from Microchip. In this case, I'm using my little work horse, the 16F88, running at 8 MHz.
Here are some photos of the rig on breadboard:
The Full Rig With Ruler
Closer View of the Board
Very Close View of Connections
I'll whip up a schematic soon.
EDIT:
And here it is in somewhat rough shape:
Saturday, January 14, 2012
GPSIM - Working Again
Awhile back, I upgraded my OpenSuse Linux distribution to the latest ( 12.1 ). I won't go into the utter disappointment that is Gnome 3. I got most of what I needed working in short order, and I reverted to the Gnome "throwback" mode. What I couldn't get working was gpsim. It depends on gtk-extra, and it seemed to need the ancient 2.1.12 version. Here's what I did to get gpsim working. I ended up using gtk-extra-3.0.1.
Linux kernel: 3.1.0-1.2-default
get readline devel package. I used version 6.
get and install gtk-extra-3.x. I used gtk-extra-3.0.1
get latest gpsim from svn. I'm using gpsim 0.26.99
alter the configure file:
Mine was around line 4880.
Any lines that reference gtkextra-2.0, change to gtkextra-3.0
Mine look like this when done:
if test "$use_gui" = "no"; then
X_LDFLAGS=`$PKGCONFIG --libs glib-2.0 gthread-2.0 gmodule-2.0`
X_CFLAGS=`$PKGCONFIG --cflags glib-2.0`
Y_LDFLAGS=
Y_CFLAGS=
else
$PKGCONFIG --exists gtkextra-3.0
if test $? != 0; then
as_fn_error $? "Cannot find gtkextra-3.0 package" "$LINENO" 5
fi
X_LDFLAGS=`$PKGCONFIG --libs gtk+-2.0 gthread-2.0`
X_CFLAGS=`$PKGCONFIG --cflags gtk+-2.0`
Y_LDFLAGS=`$PKGCONFIG --libs gtkextra-3.0`
Y_CFLAGS=`$PKGCONFIG --cflags gtkextra-3.0`
GTK_VERSION_T=`$PKGCONFIG --modversion gtk+-2.0`
echo linking with gtk-$GTK_VERSION_T
.
N.B. - Don't change the lines that reference gtk+.
execute:
$autoconf
$./configure
$make
$su
Password:
#make install
Now, for OpenSuse ( and other distros, I'm sure ), the default search path
does not look in /usr/local/lib. So, I edited /etc/profile as root, adding
this line:
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/local/lib"
Done!
Wasn't that easy? It wasn't for me!!!
Linux kernel: 3.1.0-1.2-default
get readline devel package. I used version 6.
get and install gtk-extra-3.x. I used gtk-extra-3.0.1
get latest gpsim from svn. I'm using gpsim 0.26.99
alter the configure file:
Mine was around line 4880.
Any lines that reference gtkextra-2.0, change to gtkextra-3.0
Mine look like this when done:
if test "$use_gui" = "no"; then
X_LDFLAGS=`$PKGCONFIG --libs glib-2.0 gthread-2.0 gmodule-2.0`
X_CFLAGS=`$PKGCONFIG --cflags glib-2.0`
Y_LDFLAGS=
Y_CFLAGS=
else
$PKGCONFIG --exists gtkextra-3.0
if test $? != 0; then
as_fn_error $? "Cannot find gtkextra-3.0 package" "$LINENO" 5
fi
X_LDFLAGS=`$PKGCONFIG --libs gtk+-2.0 gthread-2.0`
X_CFLAGS=`$PKGCONFIG --cflags gtk+-2.0`
Y_LDFLAGS=`$PKGCONFIG --libs gtkextra-3.0`
Y_CFLAGS=`$PKGCONFIG --cflags gtkextra-3.0`
GTK_VERSION_T=`$PKGCONFIG --modversion gtk+-2.0`
echo linking with gtk-$GTK_VERSION_T
.
N.B. - Don't change the lines that reference gtk+.
execute:
$autoconf
$./configure
$make
$su
Password:
#make install
Now, for OpenSuse ( and other distros, I'm sure ), the default search path
does not look in /usr/local/lib. So, I edited /etc/profile as root, adding
this line:
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/local/lib"
Done!
Wasn't that easy? It wasn't for me!!!
Subscribe to:
Posts (Atom)