Software Engineering Week 8 - The end

Course Summary

This course was a solid stream of software engineering concepts and applied work. There are two major takeaways for me:

1) People matter, a lot.
A theme that our instructor worked to get across was that people trump process. I am personally attached to the agile process because I do not know what it is like to work for people with high levels of planning. My career has been spent in small startup companies and generally chaotic environments where we design and deploy as quickly as possible. So that experience has probably misled me to believe that the process trumps the people. Generally I have been lucky to work with good people though. If I hadn't had such great people to work with then the process wouldn't have made things better.

In this class I was lucky to be assigned to a friendly and positive team with a good outlook. They communicated with each other and were not quick to get offended. So we could propose changes to the design, implement features, and work together without a lot of friction. This class was short and so was the team project but that only reinforces the point that people matter, a lot. With the wrong people this could have been a very negative experience due to the opportunity for stress presented by the timelines involved.

On the other end, the instructor for this course is highly focused on individual development. That aligns with my personal interests. I don't actually care what grade I get in any class, or if I pass, although that is a bit of a hassle. There isn't an end to learning for me and I could learn a lot if I had to retake this course. The problem for me would be taking a course with the wrong instructor and classmates. After all, people trump process.

2) Time spent on design is time saved programming
I have never been a heavily design oriented person. Generally I hack my way to success and figure out what needs to be done as I go along. This process presents a problem though: to scale up and delegate tasks to other team members I need to focus on learning the skills needed to share design ideas. UML for example is a great way to create a variety of diagrams that can convey your vision before work begins on a project. As work continues you can update these diagrams so the team has a shared vision. Without enhancing my skills in this area I will be limited in the complexity of projects I can engage in.  Software design is a core area for me to work on long after the course is over.

Software Engineering Week 7 - Getting Organized

Week's notes

This week was mostly about moving along on the final exam. I broke that up into a few nights to have time to think about things and go through the problems. It took a good amount of time to make things nice and there was a lot I didn't know before taking the test. Java interfaces for example was something I hadn't looked at much before. Interfaces provide a nice way to follow a pattern for your classes and the best article I found was on tutorials point of course. They seem to have the best info out there. The reason I had to look up interfaces was because of a question about using a publisher/subscriber pattern. The big thing I know about pub/sub is MQTT, a protocol for publishing and subscribing messages. 

Aside from the test I need to get back on our team project. We stopped before doing our first refactor. Without moving the gamestate out of the main game loop we can't call ourselves a view/model/controller based program. So there is some more work to be done on that. I still haven't figured out JPA and how to get Java to work with MySQL though. That is my big concern for reaching assignment criteria, but more importantly for doing something interesting. Getting a good grade is less important than learning how to do interesting things. 

Software Engineering Week 6 - Getting Organized

Weeks Notes

This week was about getting our project up and running, deployed online, and managed with pivotal tracker. One team member took a clear lead in developing the base software for our game. I scoped out what was going on and updated pivotal to see where we could go next. It was a fun week for sure but a bit stressful. My concentration has been on the team project and not really on gannt charts which was another topic for the week.

The content about gannt charts makes me happy to have friends who enjoy project management. I hope to continue to work with people like that and avoid this kind of organization as much as possible. In contrast this pivotal tracker program for Agile projects is really fun. It is nice to set goals for the week and be able to comment on them or move them around with ease. That kind of project management is no problem for me.

Using Pivotal Tracker and figuring out how to compile, debug, and upload the style of Java applications that this class uses has us at a nice start. Things are almost over though so that isn't great. I would like this class to be twice as long and have more fun with our application but that isn't reasonable because of the way this program is setup. The alternative of attending classes in person isn't an option for me either though due to work. So I think the 2nd best, and most expensive option is to just take classes like this a few times so I produce usable products.

Looking ahead

I still haven't been able to get a local JPA application working with MySQL. So that is the big problem for the next week. If my team gets it working on the server that is fine for the project but I need to be able to duplicate anything locally. By being able to run, debug, and modify the program locally I can do a proper code review and provide useful feedback for where our software can head during the next iteration. We also need to do a refactor to get our software away from everything running in the main Java applet. It would be ideal to have the applet handle the requests but have game objects that keep track of game state and let us have a game loop off to the side.

Software Engineering Week 5 - People Solutions

Week's notes

The focus for this week was on interacting with others and the importance of reviewing each others code. Based on the reading it seems like the agile process supports better code review than waterfall. The ideal code review seems to be around 60 minutes long for around 3-400 lines. Weekly short code reviews allow people to find errors without getting burnt out. If your development cycle is based around 1 week intervals with review and refactoring going on then there probably aren't going to be huge new chunks of code that you haven't reviewed any of before. That seems ideal for maintaining code quality. 

Aside from code review there were tips on how not to offend people and make things go smoothly. This doesn't sound like as much of an issue to me because these things work themselves out. If somebody is really terrible with people then they probably have some kind of social disorder. On the other hand they could just be rude. Ideally they have some kind of social disorder which you can figure out to become great friends with them. If the problem is that they are just rude then you won't end up working with them. Recruiting and management should factor those people out before they cause too much trouble. So advice on working with difficult people isn't really that great unless you're trying to be a salesman. The best advice for "working with difficult people" is to avoid organizations with poor management and recruiting so that you don't have a terrible day. Otherwise you are wasting your time and health.

In the background the homework continued learning about interesting Java technologies with applications in certain commercial fields. I'm still trying to get all of this going. MySQL is setup and I can use the workbench that came with it to load up a basic table. The problem I am running in to is related to running the JPA program that actually connects to the database running on my local machine. It's also still a bit tricky keeping track of how all the different Java things link together, what they need to run, how they all run, etc. The technologies seem useful though and established as a standard way to build applications. It does seem like a good deal of repetition and study would be needed to utilize the whole Java webapp route. 

  

Looking forward

Moving ahead we are ramping up on our group project. A basic repository is setup and game frame work is being built. The big trick here seems to be getting going. It is important to get a working base going that we can grow off of. Without that base then you could end up in a bit of a traffic jam having everybody trying to do the same thing. So I look forward to us branching out this week and then selecting and hitting targets with our project management software. 

Software Engineering Week 4 - Don't get hacked and do your homework

Week's notes

This week was pretty good. It feels like it is a bit slow for me to absorb the information but I enjoy the current course material. I am trying to learn a lot of new things in different areas at once so that can make my learning velocity feel like it isn't good enough. This week settled some problems for me by focusing on being device independent so I can do my work anywhere. I can have a different device for software development, design and documentation, and use other devices for course assignments like this. 

 As far as learning for this week, after trying out a few different programs for making UML diagrams my favorite is a google docs plugin called draw.io diagrams. The important thing for me is that I can access my files from any computer running windows, Linux, or chrome OS. That way I can spend more time on my course work. When things are locked to a single platform or require software installed on every device then that puts a wrench in things for me. 

Aside from process improvement there was some good reading this week comparing hacks of Sony and Target. The important thing for me from these articles is that inside the designer facade of a companies brand there are just people working at their computers. Every company isn't filled with super hackers designing their systems. Sometimes people just want to buy some software from Microsoft and get things up and running. Then they end up getting their systems broken into and lots of account information leaked. 

I don't fall into the category of people who say this is inevitable. If somebody is really motivated they can hack any system right? If the system is well designed then it gets harder though. Exploiting software and protecting against unwanted software use it isn't an area I know much about though. The important thing for me to keep in mind is what I don't know with regard to security.  By doing that I can work to avoid setting up critical systems without consulting experts. Also by utilizing known libraries when possible instead of getting creative with security schemes I think I can help to avoid introducing new opportunities hackers. 

Upcoming work

The exciting part for this week is ramping up our group project. Thanks to the selection of the Agile process for this project we can really make something fun. By having the project open ended and with a short weekly cycle of design/develop/deploy we can make something interesting without stressing out too much. 

Software Engineering Week 3 - UML could work out

Week's notes

The focus for this week was running into problems with a bunch of Java technologies to prepare for our upcoming group project. If we ran into these basic problems in week 5 then we would be out of luck. Instead we should be able to make a deliverable over several iterations. This week by making a hangman game with servlets and jsp files we demonstrated the key pieces needed to build a web app as a team. Key problems included loading two resources types from the server to display in dynamic pages and saving different game states per user. User input was also included but that was in earlier homework so not super critical.

UML  - A bunch of different types of diagrams with one language

At first glance it might seem like UML is going to be some niche way of making software diagrams, like using google drawings or custom drawing apps. Before you know it maybe you're up and running trying to make diagrams with UML to solve problems without knowing what's really up. That's how I felt until seeing that UML could be really useful. The surprising part to me is that there are so many diagram types that are all written in this language. Until today looking at these diagrams I wasn't sure what was going on and a little frustrated when new diagrams that claimed to be UML looked so different from each other. I think this is due to a resistance to be sold on a new keyword or language until it settles in. 

Design Patterns - they have some names

Remembering the names of design patterns seems pretty tricky. My interest is in the shapes that you see in software and matching the patterns to have a clean flow. So I look forward to studying more of these shapes and looking at sample design patterns. 

Software Engineering Week 2 - Test it all

This week I had some problems that threw me off a bit. So I didn't submit anything really. I did learn a lot from the reading and videos though. From those I can tell that big waterfall style software projects are not really my interest. Listing out how everything is going to work seems like a job for a different personality type than mine. What I like to do is lay out close targets for what can be done and then figure out where to go from there.

An example would be writing a piece of software to test an electronic device. Instead of getting all worked up planning the test software my first order of business is going to be making sure that I can control the instruments needed to interact with the device. This would result in needing to get GPIB and ethernet control of the devices working. With that doorway open I'd move ahead and lay out a few more close targets to hit until I reached my goal.

I can see the case for using things like UML diagrams though. Instead of using them in a waterfall process it would be nice to share them as google diagrams and update them as the project goes on. We will do that in week 4 I believe as we ramp up our team project. Normally I just try to keep these kinds of diagrams in my head as I work on a software project. Laying them out on paper seems like a good way to keep everybody on the same page though as we collaborate in a group format.

The emphasis on test was also interesting this week since I have worked in manufacturing test for a while. Having a test for just about every single piece of your program really seems to make sense but I haven't worked like that before. It reminds me of working on a poorly documented electrical cabinet for a test system. When I was first deployed to work on this large cabinet I wasn't sure where to start. Then someone told me "just trace every wire" and I thought they were joking. Sure enough, I went ahead and spent weeks tracing every wire to every electrical component and documenting it on a new schematic. This seems a bit like the test route where we should really have tests written for all of our code. At first my reaction is "really, every piece??" but it makes sense. The availability of automated test suites that can run through your tests makes it appear that this would actually pay a good dividend in the end. So long term I would like to follow along with this and figure out what the proper automated test route is for a given software development environment, Java or otherwise.

So this week I lagged due to some personal problems and I might fail but that's OK with me. The important thing is to just keep at it and learning new skills. Every repeated class gives me even more time spent reading and solving problems. I don't mind being a bit slow or scattered as far as my thinking goes as long as I can build interesting things.

Software Engineering Week 1 - Off to a good start

Week 1 - Off to a good start

There's lots of different people in the world so starting a new class is always interesting. The great thing about this class is that the way it is being run makes sense. There's no weird stuff, no tricks, things are just laid out in a way that looks well designed for student skill growth. What are the key attributes that appear so far which lead me to believe this?

1) Course content available early - If you are disorganized like me and only looking to focus on critical tasks as they pop up it is important to have as much information as possible. A friend once told me to ask the question "what happens if you do nothing?" as a metric for task priority assignment in a resource constrained environment. With more information available I can find key targets that can be hit early to prevent stress if

2) Consistent forum posts leading to e-mail updates on my phone for core assignment road blocks - In a class like this there are going to be generic problems that everybody runs in to. The instructor and TA in this class seem well aligned with the tools at hand and quickly respond to growing problems early enough during the assignment week for resolution.

3) Open ended assignments - This is a critical attribute that seems to appear in well run art classes. By leaving assignments open ended there is room for people to focus on the areas of a problem that are inspiring to them. As an example in an English class of mine there was extra credit for making a video about a writing topic. So I went ahead and produced a short cartoon on my computer, loaded it on to a VHS taps, and played it in class. The other students weren't sure what to make of my cartoon with a dinosaur eating people but the skill growth was there for sure. Similarly in a software class like this there is room to throw in a cutout of a dinosaur or anything that makes you happy and doesn't mess up your agile schedule.

There are still plenty of ways to mess this up for sure. It won't be because of poor course structure though. Personally I look forward to seeing what cool projects people create within this framework over the coming weeks.

Service Learning Week 4 - Capstone Ideas

Capstone Ideas

The main assignment for this week was to come up with some capstone project ideas. I lucked out because I try to keep a good buffer full of project ideas like this. The difference of course is that I generally don't complete them because I am jumping around from one thing to the next learning new techniques and only working on larger projects at work. So what are my 3 main ideas?

1) Mesh networking solar garden light: I have played around turning jars into mesh networks before but never finished a solid garden light. So this project would be turning my concepts in to reality making a high end garden light.

  '

2) Multi-sample colorimeter: This device would shine light through tubes of liquids and measure how much of a given wavelength is absorbed. It isn't quite a spectrometer, but is a simple useful device for measuring samples of fish tank water for ammonia, nitrate, and nitrite. The icing on this project cake is that it would have wifi or bluetooth to log the data through your phone or home network (undecided) and hold multiple samples. So each colorimeter in the array would be used for a dedicated sample like ammonia measurements. Then you can shake up each vial and put them in their dedicated slot for sample measurement after the sample settle and changes color. 

Example open source colorimeter by IO Rodeo

3) Mobile application for partkeeper inventory software: Partkeeper is the best open source inventory management software I have come across. It isn't a manufacturing resource planning software. It also isn't an enterprise resource planning software. It is an inventory management software. This is a key distinction because a lot of the web based solutions to this problem are offered as "software as a service" with a monthly fee and have way too many features. For managing labs like I am interested in that is excessive and distracting. My interest is making sure that we have the proper components in stock and that we have a reference component list for generating new designs. 

The part that is missing from PartKeepr is a mobile application for rapid inventory assessment which would be my project. Finger worn, battery powered wireless barcode scanners like the RS507 are available to interface with a smart phone via bluetooth for taking quick stock. This kind of application would allow for workload segmentation by dividing ordering and stock assessment between different individuals. As things are now generally a lab manager will take inventory and restock parts in a tedious process. This process should be streamlined to include wearable and smart phone technology.

RS507 finger worn bluetooth barcode scanner for rapid aquisition of product identification codes

 Other news

In other news we continue to learn about social justice through a steady workload of reading and videos. In this area I think the primary way I can make a contribution is through assisting others with technological skill development. On a day to day basis that consists of short conversations where I give others key information required to continue their skill development. These conversations can also include quick tutorials where I show people how to do things like design simple parts in 3d CAD software like SolidWorks. By getting people in the door designing very simple mechanical components or printed circuit boards they can kickstart their learning and get in the game.

Service Learning Week 4 - Just figure it out

Project Management

This weeks key question as part of the service learning class is regarding project management. My background in project management is focused on engineering and gardening. They might seem like very different areas but they have a lot in common. In gardening when I move to a new home the first thing to settle is my fruit trees since they will take the longest. Then I plant my berries so they can reach out and grow into large bushes that produce year after year. Finally come the vegetables, since their lives are measured in weeks.

For a hardware engineering project, using a firmware download station as an example, you want to verify functionality as soon as possible. This means requesting a sample printed circuit board assembly ASAP. Without a sample board you are going to be out of luck, and who knows where that is in manufacturing. People can also be busy and slow to respond so critical human interactions should be loaded to the front of the que. Then you can begin your fixture development based on the presumed correct PCB layout files. A repetitive cycle of e-mails should be sent of course to ensure you get the physical components you need from the PCB designers. Then you begin your 3d prints while still working on your fixture design so you can make sure everything works with a sample board you get. From there on it should be smooth sailing because you have front loaded the critical tasks and verified your design meets the requirements with prototype components.

Then of course if you are a project management professional you can spin several plates (projects) above your head on tall sticks all at once! The end result of successful project management being a peaceful work environment where the critical problems are loaded to the front and available technical and personnel resources are utilized efficiently.

Service Learning Project This Week

Why the title "just figure it out"? This is a classic saying that I have always told my friend, a bit to their frustration. It's true though, you can just figure it out. Computers and electronics are designed by people so they make sense. You may not understand it at the beginning of a problem but you can absorb it. That means if you just remain calm and work at it bit by bit your odds are pretty good at solving your problem whatever it may be. 

Take the LPX-600 3d scanner by Roland found at Monterey Penninsula College (or even the LPX-60) for an example. This scanner "does not support" 64 bit windows 7, windows 8, windows 10, anything past windows XP it looks like. Reading that is pretty frustrating if you'd like to use it with a newer computer! So relax, think for a second. What does it mean that this scanner is not supported under these operating systems? I tried installing the control software and that installed fine. What didn't work was the drivers. So you can look on the internet and download a virus from some Russian website, or look in the drivers folder and see what it really is. It turns out that this device uses drivers that are made for a FTDI communications chip. Conveniently enough they even still update these drivers.

With new FTDI Virtual COM port drivers the only thing you need to do is copy a few lines from the drivers supplied by Roland to add the device information to the new driver information file.[Instructions on github soon] Then you can use the device under windows 7 64 bit and probably newer operating systems supported by FTDI as well. 

Overall my point is that people need to relax, think things through for a second, and keep moving up stream. Maybe there is a Linux version of the software that solves your problem? Maybe you can figure out what chip is inside a machine that you are actually talking to? Whatever the problem is if you don't freak out and just keep at it you can figure out any problem that you need to. 

Service Learning Week 3 - Geometry for manufacturing from 3d scans

A common problem with 3d scanning is that the geometry you get from it isn't great for manufacturing. 3d scanners generate a cloud of individual measurement points which are converted into a mesh of triangles. That's all fine and dandy if you want to 3d print a part or render it in a computer generated scene, but what if you want to generate some real parametric design files that you can modify easily in a program like SolidWorks? Also what if you want to prepare the models for manufacturing on a 2.5 or 3 axis milling machine? Well that's the topic for this week.

The solution comes in reverse engineering software like Geomagic which Monterey Penninsula College are currently evaluating. Geomagic mixes aspects of 3d scanning software with aspects of CAD software like Solidworks to allow for easy tracing of models. That converts them from a cloud of points or mesh of triangles to a clean structure of shapes like circles with defined radii, fillets, and other parameters that are easy to modify.

To see what this kind of workflow might be like I setup a trial version of geomagic on a PC at MPC. Then I scanned an old pulley with their high end roland 3d scanner. This point cloud and incomplete mesh was imported in to Geomagic which allows you to perform operations to close gaps, smooth bumps, and trim excess data from the raw scans. With that work complete you get a finalized sealed up mesh which can be used for tracing.

Tracing the general pulley shape from the side created a new model of the pulley which is the kind of geometry you need for regular CAD programs that are for mechanical engineering instead of art/sculpture. The pulley could then be imported in to a large assembly, modified to fit different belts or shafts, and manufactured with CAM software. Overall Geomagic provides a great solution for converting these kinds of models and I look forward to examining the software more and providing a purchase recommendation to the school. 

Service Learning Week 2 - It's go time

Service Learning Week 2 - It's go time

Lights, cameras, and action...these are all things that will be found in my upcoming documentary. Also to be featured are lasers. Some of these lasers will cut things, some will convert physical objects into 3d files on a computer, and some will turn liquid in to solid forms. That's for the future of this class though. This kind of a blog post should really start with the present or the past. 

So what was this week about? Well I went to MPC and checked out all their equipment. After spending a good amount of time going back and forth between "wow this is crazy" to "man you guys are really going places" we settled down. With the excitement at a manageable level we commenced with assembling and testing their new stereo-lithographic Form2 3d printer. Then we downloaded some parts and send the printer off on it's way for a 5 hour print. This part was quick because we have had several Form printers at my work which make great fixture components. 

With the printer running at full steam shooting a laser into a box filled with a liquid resin to harden it I jumped on the vinyl cutter. This machine had some driver issues which were quickly resolved. Then I learned about all the things you can do wrong when operating the machine. This led to eventual success with a fully setup machine loaded with blue vinyl and ready to cut out some pokemon logos. The key learning take away from this process is that computer controlled vinyl cutters let you make very large quantities of stickers for very cheap quickly. They are also ultra simple to operate after you make the first few mistakes. 

Aside from awesome machines there was a bit of paperwork for the class. I don't mind this paperwork though because it is not as rigid as other classes. It is a lot of reflection about your understanding of the world, how you fit into it, etc so that you can help make it a better place. If it was more rigid I would be a bit out of luck because that hasn't worked out for me traditionally. So thanks to the people involved in structuring this! I'm pretty sure you are the only people that will read this post so have a good week as well. 

Service Learning Week 1 - Getting everything in place

Week 1 - Getting everything in place

At the beginning of this week I thought I was doomed because my service opportunity was agreed upon but the details weren't really fleshed out. A few text messages with my friend and everything was locked down though. Monterey Peninsula College has lots of work to do setting up new equipment and software to get ready for the fall. I'll try to be there once a week because it is a long drive to get there from my house. When Saturday is not an option for my service learning partner then I can take time off work. Overall this is great though and basically what I have been doing for many years. Selecting equipment, purchasing it, installing it, learning it, training others on it, and integrating it into peoples work flows is awesome. Adding new capabilities to this community college will have a long lasting impact on the community. 

With the stress of figuring out my service opportunity melting away into excitement of machinery there was another interesting aspect of this weeks assignments. The other fun assignment I had was to write about the history of my family and technology.  To do this I went to my parents house and sat out on their patio with them. We started at the beginning with my dads first computer, through our first e-mail, and up to my collection of manufacturing robots. It was a really great experience. Combined with our weekly reading though it made me really appreciate my life. I have had a uniquely consistent opportunity to learn and use high end technologies before most people know they exist. This has shaped me as a person and I hope to continue to share the skills I have collected with others.

Intro to database week 6 - querious questions

Week 6 - querious questions

This week we went back to SQL with full steam. There were your run of the mill MIN() and MAX() type queries, but also a lot of sub-queries that took me a bit to figure out. The syntax that SQL wants for sub queries threw me off a bit but it seems like you can really do a lot stringing things together. You can also write things that may seem nice to you but turn out to look absolutely terrible. So the big take away for me was to not make a mess, like with any software project. Nobody cares how smart you think and clever you think you are if your query, script, program, whatever looks like junk. It's just not useful. 

Here is an example from the homework that I like, for figuring out who the best student was in a given course code. 

You could write that query a lot of ways, including one giant line that is a big hassle to read. But as written above it is clear what is going on. We are interested in the last name of the student we will call the best student. Their name is in the student table but the grades are in the enrollment table so we will join those tables together. The unique identifier of interest is the student number so use that for the "on" condition. To round it all out we are looking for the best grade, that's where the sub-query formatting comes in. It is broken into "where a > b" but we can expand the b part out by putting parenthesis around it. That lets us chill out and start over from the beginning with a query that figures out what the max grade was for class # 10014. When it all runs then we end up with the student with the highest grade from class 10014. 

I still don't know what I'm doing as far as SQL goes but at least I know what to look for and the structure of it. To really be able to get much done I think you need a lot of weeks of repetition on real world problems where you fiddle around for a while. It is clear that sub-queries really let you start doing useful work instead of just finding simple pieces of information. 

Intro to database week 5 - What the graph?

week 5 - What the graph?

This week was a bit jacked for me. I was supposed to mail a box to Mexico but then got a call "we'll let you know in a few hours if you need to mail that stuff". Instead they called and told me to go to Mexico the next morning. What was interesting about that is that my battery in my laptop sucks so I wrote on paper this week. I don't normally write on paper at all during the week or month. It was nice to try that out for a change. Printing homework, and writing on paper, pretty odd. Then I spent a while learning some online diagram software along with diagram software for my phone so I could make things look nice because my hand writing is really bad. It looks a bit like I'm having a seizure when I write but my hands are actually pretty steady because I can do microscopic soldering for circuit boards. 

That's water under the bridge though because my homework is in and this week was really great for me overall even though I was so slow. After getting home from Chihuahua I watched a good documentary on Sunday about the guy that made the Segway and he said he's a really slow learner but he just keeps re-reading things until it soaks in. That was my plan for this week and it seemed to work out. So overall this week the subject was cool graphs that are your pre-planning for database development, but also an iterative process during database design and implementation. Drawing graphs that show how items are connected lets you figure out where you are going to run into trouble or see new connections you didn't think of before. Two kinds of graphs were presented, the Chen model which seems like it isn't very cool, and the Crow's Foot Model which seems very nice indeed.

What makes the Crow's Foot Model So nice? Well for one it seems that good graphing software supports it's symbols and makes creating these graphs fun. These diagrams also are very descriptive in expressing the quantity of each item and their relationship. Are there ten sailors per ship? Does each sailor get their own ship? Is there a law that says people can only have two kids or one car? These are all important information for your database that look like garbage when written out in long sentences. They of course need to be compressed into nice connected graphs that are concise and show you what is going on at a glance. Nobody has time to read your epic novel about rules and by requiring interpretation you leave room for mistakes during implementation. So draw cool graphs instead.

Well, that's the critical information from this week summed up. Draw nice graphs like this one homework 5.3!

week 5 pt 2 - Multi-frequency & multi-sample smart colorimetry for water analysis 

The big distraction for me lately has been fish tank water. What's a nice way to press 1 button and have information logged to your database about the quality of your fish water? Colorimetry is popular of course, like this example unit from IORodeo .

There are also a variety of commercial handheld units with a specific frequency of light that they use to measure things like chlorine and ammonia. The basic principle being that they measure the absorption of light of a given frequency going through a sample to determine the concentration of a known solution. For example with fish water there are sample kits from companies like API which let you test a variety of chemical levels. PH, Ammonia, Nitrate, Nitrites, these are all critical information for keeping fish healthy and happy. Low cost colorimeters generally hold one sample though and use one frequency of light. They also don't seem to automatically log the information to a centralized database. It's almost as if they think you're going to want to connect the device to a computer over USB and log the data yourself, or worse yet write it on paper. As we discussed earlier that just isn't an option. 

So it seems to me the solution is clear, the world needs a low cost colorimeter that hold many vials at once for testing difference chemicals simultaneously, which also logs to a database on it's own. It would probably need some internal storage in case you are lazy and don't connect it to the internet, and it should be able to connect to your smart phone of course because everybody loves phones. I think this could be good project because although my interest is fish water, my dislike for repetitive actions could benefit others by providing an inexpensive multi-sample solution. By allowing the device to be re-configured with a phone application each sample could be tested for concentrations of a different chemical, or multiple samples could be tested for the same chemical. This would require a few light sources for each sample so that the wave length projected could be customized. The good news there is that LEDs are just about one of the least expensive things in the world right now. 

On the receiving end light sensors, even advanced ones that give you lots of information, are also very inexpensive these days. Wifi enabled microcontrollers like the ESP2866 are also very cheap. So it seems there is room in the budget to make the housing look really cool and have a unit that holds 4-5 samples at once for testing with the press of a button.

Intro to database week 4 - Keep in Normal.

Week 4 - Keep it Normal

This week was about Database Normalization. This sounds like a tricky subject but it is really just the process of figuring out how to set up a database that isn't a trap. That process involves figuring out what should be in each table and how the tables connect so that you don't end up with redundant data or paint yourself into a corner as far as flexibility goes. For example: would you mix customer data and product data in the same table? What happens if you do? Well there's fancy database terminology for 1NF, 2NF, 3NF, and 4NF but what really matters is that you will make a giant mess. That's what I got out of the end of this week. 

Sherif Ramadan has a good video on 3NF which is really the form we want things in for most cases. 

Figuring out how messed up your database is consists of drawing out dependency diagrams. These let you see problems such as partial dependencies and transient dependencies. Look at this example from our homework relating to prescriptions.

The key for this table is MedNAme, PatientID, and Date. But what does a patients name have to do with the date? Also what does the patient name have to do with the medication? Well the patient is taking some medications, but if you just have the medication name and the date then that doesn't determine a patients name. A big hospital will have lots of patients taking the same medication on the ame date. The patients name is determined by their patient ID though, so this is a partial dependency for this table. That makes you start to think, maybe I should make a table with just patient information? That would make things nice right? The same goes for medication names and their shelf life. Why would you have the shelf life of a medication in a table that is discussing patients names? Of course that's odd, without even knowing about 1NF, 2NF, 3NF etc you probably are starting to think it makes sense to have a table with just medication names. Those are both examples of partial dependencies. 

Transient dependencies would be when an attribute that isn't part of the key determines another attribute that isn't in the key. Look at this example table below regarding invoices which hasn't been normalized. Vendor code determines vendor name. So why is all that information in one row the table invoice? Obviously you should get the vendors their own table and do the same for the products as well. Then you can have the product table use the vendor codes as a foreign key so you know who to buy your products from. 

Overall it seems like the goal of normalization is really to drill into your head how to organize things so that you don't get into a trap. If you start writing lots of code to use your database and don't take the time to really think things through you will spend time back tracking or just end up trapped when you try to scale. So this is a great section to get us going on the basics which will eventually just become a kind of gut feeling for bad design.

Intro to database week 3 - SQL gets real

Week 3 - SQL gets real

Up until this point we've mostly been talking about the theory of databases and how they generally work. This week was nice because we started to chug along on a real database. Since I have no experience writing SQL commands the first thing I did was hit the ground running by downloading an SQL training app on my phone so I could do problems on BART. 

My phone runs Android so there was a nice knowledify SQL app to run through lots of practice problems. The video below shows how this app works if people are interested. This app is nice for a few reasons. First of all it has auto-complete so you don't feel like you're wasting your life typing things out in full for no good reason. Overall the user interface is very nice and it has a quick guide to let you brush up on basic commands as you learn them. The app looks like it runs out of questions pretty quickly but I think it is a good free way to get some repetition in for someone just getting started like me.  

With basic SQL queries in hand the week's homework was filled with setting up example databases and querying them. I think if we stopped here I could do some useful things in my side projects. For example just with what we learned in this weeks homework we can download the cx_Oracle python library and start parsing files and loading the information into a a database. This let's us scrape data from social media sites to harvest information about people and automate tasks based on that information. Without the information from this weeks topics I would be storing that information in a comma delimited file which is a bit of a mess. Having the information in a database makes searching for data like last profile view data simple in python. For example after harvesting used id links you could search for pages you haven't visited yet by doing this:

import cx_Oracle
db = cx_Oracle.connect('linkspider', 'password', 'localhost:1521')
cursor = db.cursor()
cursor.execute("select User_ID from linkedINUsers where LastViewed is null and rownum < 10")
userIDs = cursor.fetchall()

cx_Oracle wants you to get a cursor object which let's you act on the database after you connect to a database and get a database object. From there you can use the cursor to start performing the commands that we learned this week. Select rows, modify them, add new ones, etc. By using rownum in your query you can limit the results returned because if your bot tries to view too many pages too quickly you'll get caught! 

Before you send your bot out to start harvesting you'll want to get some data of course. The example I wrote below shows how to do that for a basic linkedin html file taken from the people you may know page. By seatching for "/view?id=" we can scrape a list of user names and store them in the database. The critical part here is that you must perform a commit on the database. Otherwise your results won't be stored! You also probably don't want to waste time trying to commit duplicate user ID's, so it is nice to keep track of them as you go along. Another important aspect is that when you try to add rows to the database that already exist then cx_Oracly will throw an integrity error. So make sure to catch that and do whatever you like. 

import sys
import cx_Oracle
db = cx_Oracle.connect('linkspider','password', 'localhost:1521')
cursor = db.cursor()
sourceFileName = sys.argv[1]
sourceFile = open(sourceFileName, "r", errors='ignore')
buffer = sourceFile.read()
index = 0
done = False
urls = []
OK = 1
matchString = "https://www.linkedin.com/profile/view?id="
matchStringLen = len(matchString)
while OK != 0:
   index = buffer.find(matchString, index)
   if(index == -1):
      OK = 0
      continue
   index += matchStringLen
   index2 = buffer.find("&", index)
   match = substring = buffer[index:index2]
 
   duplicate = 0
   for url in urls:
     if(match == url):
           duplicate = 1
   
   if(duplicate == 0):
      urls.append(match)
      try:
        cursor.execute("insert into linkedInUsers values('" + match + "', NULL, NULL, NULL, NULL)")
      except cx_Oracle.IntegrityError :
        print("Already in database")
       
   index = index2
db.commit()

Overall I think this was a solid week. We learned some really useful tricks to create and use databases. Now we just have to make sure to keep going otherwise we could end up thinking we know what we're doing, which would end up creating a real mess! At least now we could do basics like monitor home automation systems, aquaponics system sensors, and other simple projects. Just don't go off trying to apply for a database administrator job at a bank! 

Intro to databases week 2 - A relationship with Algebra

This week was serious meat and potatoes business: relational algebra and some SQL.

I know what you're thinking: what's relational algebra? Well it's like your logic & computation class met SQL and decided it didn't want to have a good structure and be a real language. So relational algebra gives us some example ways to perform operations on database tables in a bit of pseudo code. Useful keywords in this area include the following including simplified description:

SELECT - pull some columns out of a table.
PROJECT - pull some rows out of a table.
UNION - join two tables that share columns
INTERSECT - join two tables that share columns only on the rows that match
DIFFERENCE - subtract one table from another, giving you the first table with the values in the second removed from it when they match
PRODUCT - makes a super table that is the combination of the two multiplied out in almost every combination. it generated a bit of a mess but is used as part of the join operation.
JOIN - add some columns from one table to another to make a super table where the keys match.
DIVIDE - this one is kind of weird and hard to describe without a picture, but it's included for keyword completeness.

After getting all the keywords loaded up we had some exercises to do. Those exercises really served to provide the syntax for using relational algebra as if it was SQL in the final homework for the week. That seemed a bit akward like we should have just finished loading up SQL commands and then wrote it like that. But it is what it is.

From there we went on to reading but not much homework related to SQL. That's where it gets to the serious business part with real structure to send commands to a DBMS.  The big take away for me from this weeks section on SQL is that it seems like something that is straightforward to get the basics of but then tricky to become really good at. Being able to ask questions of a database is fine but the trick seems like it will be figuring out what questions to ask and how to organize the data to generate useful information. Then there is the ever looming problem of creating a database that turns into a mess so it seems important to follow along closely with these beginning sections.

Next week should be interesting as we continue our exercises but start to use an Oracle DMBS to run databases on our own computer. I am looking forward to the hands on work coming up.

Intro Database class week 1!

Hello internet,

This week's in-school learning was really about pre-loading students with all of the important vocabulary necessary for us to discuss databases. There were different key types, integrity between tables, and lots of examples of how you can go terribly wrong by knowing "just enough to be dangerous" with databases. I think the most important aspect of this was that just because you know how to use some database management software that doesn't mean you know what you're doing. Worse yet, if you actually convinced people you knew what you were doing you could create a massive time suck that becomes difficult to maintain. Hopefully though students don't head off on their own immediately and convince someone they can run a database.

From the homework we saw the importance of unique keys to identify rows within a table, and how to neatly link tables together with those keys. Things like SKU's on items in a store, student ID numbers, store codes, and all the other random numbers that don't seem to really give any info become magic numbers inside the database. This of course depends on being smart about how you set up your tables and which information you put inside of them. For example you probably don't want to list names unnecessarily when you can refer to people by number. Although they might feel a bit alienated during conversation, this actually ensures that you can update their name in only one location and have it updated correctly in all tables, without creating redundant names that have different spelling.

So overall I think it makes sense to continue this course and learn how to make useful databases instead of just trying to convince people I know a few key words related to database development. After all, it doesn't really matter what people think you know unless you're a shifty salesman. If you want to be part of an agile team that can manipulate matter and information to develop devices and harvest information to generate knowledge then you have to be able to do things.

Also I have some kind of fever & cold this week which makes me feel a bit dumb. It's always interesting being sick and seeing how it affects your head.

Have a good week.