What I hope to learn in CS3216

As the name of the module suggested, “Software Product Engineering For Digital Markets”, CS3216 is an interesting module that will not only teach you how to form a team and build a product, but also about how to sell your product and make impact. It covers all we want to learn about from the early product brainstorm, to the product development stage, and all the way till the product launch.
Below are a few things that I expect to learn from the legendary CS3216 class:

Software Engineering

Since this is a very practical CS module, of course we can expect tons of software development skills to be picked up! Through all kinds of assignments, projects, discussions and self-learning, I wish I can keep improving my code quality as well as software development skills. Moreover, software design is also an interesting area that I would like to know more, as we are probably going to create something from scratch, so it would be useful to design it well at the very beginning.

User Experience Design

I believe the key factor of a successful software is user. We all want our product to delight our users, but a product with a great user experience would probably attract more customers. In this module, I wish to know more about user experience design throughout my study in CS3216 as we usually don’t learn about it in other modules. In my opinion, UX is not just about beautiful user interface, it’s more about how happy your users will be when using your software (Users will probably delete your app if they totally can’t figure out how to use it after they download it). I guess probably I will do some self-learning as well to know more about the up-to-date UX design trends.

Marketing

It’s not enough if you have a great product, but you don’t know how to sell it. CS3216 is such a module that we will know how to sell our product to the customers. As I am a CS student, this is probably an area that I am extremely bad at. Therefore, I wish I can know about some product marketing skills from the lecture, my awesome peers, as well as the whole CS3216 experience.

“People-engineering”

Last but not least, CS3216 is not just about building great software, it’s also about meeting awesome people :) We have a group of talented students from different backgrounds, and I am sure it will be super exciting to meet and befriend with them! I wish to be able to meet cool people, learn to work with them and handle possible conflicts, and build up my professional relationship. After all, people-engineering is probably a super useful skills that we will benefit from it for a life time.

So, that are my rough ideas about what I wish to learn in CS3216, and I am super excited now for the upcoming new semester :P


Read a book: High Performance Web Sites

Recently I just finished this book, which is named “High Performance Web Sites”. I found the book when I was looking for some advices on front-end performance.

The book gave us 14 very useful practices about how you can improve the performance of your website. Although the book was written almost 10 years ago (2007), but the core ideas are still useful now, and some of them even became the industrial standard (such as minify your JavaScript code).
Here are the 14 practices given in the book, with my comments:

  1. Make fewer HTTP requests
    Of course, if there is fewer resources and smaller size for your website, the speed will improve a lot.
  2. Use a content delivery network
    Static files, such as images, are usually suggested not served by your own server. As the bandwidth is usually limited for smaller size website, if those big images files are served by your server, then they will affect the speed of other requests.
  3. Add a far future Expires header to your components.
    The good thing about browser is it has cache! Rendering web page from cache will make your website faster. But also be careful with the cache, otherwise if you update your content but your clients are still using content from the cache, that would be a problem.
  4. Gzip your scripts and stylesheets.
    Smaller size gives better performance :D
  5. Put your stylesheets in the document HEAD using the LINK tag
    Even if it takes more time downloading the css file, the browser will render progressively so it looks like it’s faster (no white screen problem).
  6. Move scripts to the bottom of the page.
    Content below the script tag will not show until the script is downloaded, and that affect the rendering.
  7. Avoid CSS expressions.
    Less common nowadays because of the pre-processor.
  8. Put your JavaScript and CSS in external files
    One reason is you should make use of the browser cache.
    Two ideas that worth mentioning are as below:
    • Post-Onload Download - defer the download of static files when the page is fully rendered (by creating tags)
    • Dynamic Inlining - Inline some components and trigger downloading of files, also set a cookie indicating if the page is visited. Next time check if the cookie present, if yes then probably the resources are cached already and hence could improve the speed.
  9. Reduce DNS lookups by using Keep-Alive and fewer domains
  10. Minify your JavaScript source code
    This becomes a standard process now for front-end project.
  11. Find ways to avoid redirects
  12. Make sure scripts are included only once
  13. Re-config or remove ETags
    It said ETags will cause overheads, and it does not work well in a cluster of machine.
  14. Make sure your Ajax requests follow the performance guidelines, especially having a far future Expires header.

Develop a plugin for Hexo - Github Card

Background

Sometimes when I am writing a blog, I want to reference some really cool Github repos in the post. In order to do that, I usually just copy and past the link in the post so readers who is interested in the repo could just click and take a look. However, I am always looking for a way to display a reference to a Github repo or user in a more good-looking and elegant way.
There is a very good plugin that can help me do this, which is called github-cards. It’s a very small JavaScript plugin that will show a card-like widget in your website displaying github information such as user profile or user’s repo. I used it in some other project before and it worked quite ok. But the problem is writing a blog post is different than developing a webpage. My blog is built on Hexo and all blog posts are written in markdown. So there is no way for me to include a script in the post and use a div in the content.

Existing work

Then I go and search for existing solution in Hexo framework. Hexo provide a wonderful thing called tag plugin. It allows you to customize some insert special content in the post with different tags. For example, you could use

1
{% youtube video_id %}

in your post to include a Youtube video in your code.
There is one plugin in Hexo community called hexo-github, which seems to be what I want. However, it seems that its main goal is to “keep track of version difference”, and you need to specify a commit hash. But I only want to display a user profile or a repo information >_<.
Well, the good thing is I am studying computer science. So why not just create one for myself? :D If I could implement such tag plugin and insert an instance of github-cards, wouldn’t it be a prefect solution?

Create my own plugin

There is no tutorial about how to create a plugin for Hexo. So I started with what I need to implement the plugin. Basically I will need to include the script from github-cards, and also insert the div element of the cards into my post. In order to include the script dynamically in the post, hexo’s Generator extension is a good starting point. Generator can create a virtual route that can be accessed by your blog content. For example, given the following code:

1
2
3
4
5
6
7
8
9
10
var fs = require('hexo-fs');
hexo.extend.generator.register('asset', function(locals){
return {
path: 'file.txt',
data: function(){
return fs.createReadStream('path/to/file.txt')
}
};
});

We register a route to a path called file.txt that actually returns the content of file path/to/file.txt. Therefore, if we add a link to this file.txt in our blog, we will be able to access the linked file. In this way, if we defined a route to the github-cards javascript file, we achieve the goal to load javascript dynamically in out blog.

Secondly, we need to register a new tag so that we can display the card in our blog. Hexo allows us to register new tags in our plugin. Document
For example, the following code shows how to use tag to insert a Youtube video:

1
2
3
4
hexo.extend.tag.register('youtube', function(args){
var id = args[0];
return '<div class="video-container"><iframe width="560" height="315" src="http://www.youtube.com/embed/' + id + '" frameborder="0" allowfullscreen></iframe></div>';
});

It’s very easy to understand how tag works. An easy example is you simply return the HTML element to be used. In our case, we need to create a short HTML template file that includes the script and the elements we need. As suggested by Hexo, Nunjucks is used to render the template. You could also pass some parameters through the tag arguments and apply some customization.

Finally, wrap everything together. Now install this plugin and you can use it in the post as this:

1
{% githubCard Gisonrg hexo-github-card %}

Which will gives you:

How cool is that :D


Development process comparison: Codebender vs. Teammates

In this blog post, I will briefly discussed about the development process between my two projects in CS3281/82: Codebender(in particular, Eratosthenes) and Teammates.

Project set up

Codebender

One of the headache of Codebender’s project is it does not have a proper getting start guide. Before any instruction is given by our mentors, we attempt to set up the project by ourselves, but failed to do it because of some configuration errors and there is no written guide for us to refer to.
But after our mentors sent as a virtual machine box (vagrant) as well as a detailed setting up guide, we can set up the dev server easily without much setting. Vagrant provides a perfect solution for project bootstrap. It guarantees the same developing environment for all team members, and the set up script also helps to automate the setting up process. With the help of virtual machine developing environment, it eliminates the “it works on my computer” situation and make sure the development experience is the same regardless of operating system and system environment. In addition, the project has some useful scripts, such as script to automate testing and script to automate installation, which makes the setting up easier as well.

Teammates

Teammates, as a very mature project which has gone through the GSoC twice, provides a very detailed setting up guide on the Github. It covers all the prerequisites, setting up guides, what to download, how to run server and tests, as well as a quite useful trouble shooting page. A newcomer can easily follow the guide step by step and set up the project without external help. This is extremely helpful in an open-source project as it makes it easier for a newcomer to contribute to the project.
However, the setting up guide is rather long and full of text. One must be careful when handling settings as well as downloading files when setting up Teammates. I think Teammates needs some sort of setting scripts that automates the process so the developer can spend less time struggling with setting up and focus on the feature developing. Similar helper scripts provided in Codebender could be a starting point. A virtual machine environment would be good as well but it is not necessary in this case.

Development Process

Codebender

The codebender’s development process is quite standard:

  1. Create an issue (optional)
  2. Branch off from master/develop and start coding.
  3. Crate a Pull Request (PR) to submit changes and codes. When it is ready to be reviewed, label it as “Ready to review”.
  4. Code review conducted by project manager. When the PR is reviewed, it is labeled as “Reviewed”.
  5. If the PR needs changes, go back to commit new code and label the PR as “Ready to review” again.
  6. After code review passed, merged by project manager.
    As we are added as member of codebender’s repos, we can create branches as well as push changes to the main repo directly. In general, there is no convention on issue/PR naming or description. And all reviews are done by a single person (the project manager).

Teammates

Teammates has a very concise image describing their process:
process
Basically:

  1. Fork the TEAMMATES project.
  2. Submit an issue to resolve or choose an issue to resolve. Assume the issue number is 123.
  3. On your fork, checkout from the master and named your branch 123-what-you-are-doing-in-this-branch.
  4. Working on this branch. Make sure all tests passed before submit PR.
  5. Open a PR against master of original repo, named your PR as 123-title-of-the-issue. Include “Fixes #123” in your PR description with additional explanation.
  6. Wait for someone to review your PR and your PR will be labeled as “toReview”.
  7. Update your code according the review result. (Your PR will be labeled as “OnGoing” if it needs update)
  8. After code review passed, your PR will be labled as “toMerge”.
  9. Project mentor will review your code again and give your “MergeApproval” label.
    By this time, your PR is accepted and will be merged in their next release.

Comparison

So as we can see, Teammates’s development process is more standard (and more troublesome) compared with Codebender. I think this is because Teammates is a mature open-source project, and its process is more standard as other big open-source project. A standard process is a good thing as it will make your repo consistent, and the naming convention makes it easier to manage contributor’s work as well.
Codebender, on the other hand, does not have a standard process document on their repo. I think this is mainly because it is a startup product and most of the time, the contributors are the employee from the company. So they could learn the process offline. But this also set a barrier for newcomers as well.
At the same time, Teammates needs to spend more efforts on code reviewing and issue/PR management. This invisibly increate the workload for the core member as not only they need to contribute to the repo, they also needs to spend time on reviewing other’s code.
In addition, Codebender has CI set up so new PR will go through a series of checking and make sure it does not break anything. But Teammates lacks that. All tests are to be run manually, and you need a specific version of Firefox otherwise tests may failed. I think Teammates needs to work on automation of development process, otherwise I believe it’s a little bit wasteful that valuable man-hours are spent on things that could be automated (Luckily I think a CI enhancement is already working in process for Teammates).

Conclusion

After comparing both proejcts on their development process, here are some suggestions I want to raise:

  • For Codebender:

    1. Currently all repos are lack of documentation for new developers. I believe by providing some sort of developer guides, more people are willing to come and help to improve the system and that’s how open-source works.
    2. Some convention should be used for issue/PR naming. This will make the reviewer easier to know what is being fixed in this PR.
    3. Can consider have some first-timer issue as Teammates to make newcomers get familiar with the process.

    It is understandable that as a startup, development speed is an important constrict on adapting this convention/rules. But I think it’s still good to have those things in the long run.

  • For Teammates:

    1. I think we could have a automation script for project setting up. This could save some time for new contributors.
    2. We should set up CI for new PRs so tests could be run automatically. This will reduce some work for reviewer.

Why we have one-line module in node?

Recently, Node community was shocked by one guy, who removed all his packages from NPM, and left thousands of developers who used his packages in their project suffering a lot (I’ve Just Liberated My Modules). Surprisingly, one of the interesting package he removed is called ‘left-pad’, which has only 10+ lines, are heavily used by developers with more than 20k downloads per day.
Apart from the discussion on whether this developer should be responsible for his action, another popular discussion going on is do we really need this kind of “one-line” modules? These modules are usually short in LOC, and it only takes several minutes to implement for a developer with some experience. As we see the chaos caused by removing that simple “left-pad” module, why don’t we just implement them ourselves?
There are a lot similar packages like this one, for example:

  • isarray, a 5 lines module to test if the given parameter is an array in JavaScript, has more than 600K downloads per day.

Why there are one-line module?

If you wonder why there are so many popular one-line modules, I think the idea comes from a famous principle in software engineering, “DRY”, which stands for “Don’t repeat yourself”. These small but powerful functions are meant to be used frequently in software projects. So instead of creating these functions over and over again in all your projects, you should save the implementation of the function once, store it somewhere, and use it directly when you need it.
That’s why we usually have utility (helper) functions / common library in large projects, which are this kind of useful functions that are meant to be shared in different components.
Now since the best way to create such helper functions in NPM is to create a package and use it, it’s not a surprise that there are so many these one-line modules.

The reason why these one-line modules are popular at some extent is because of the JavaScript language itself. JavaScript has good parts and bad parts. For example, one of the bad parts is about type. JavaScript’s type is easy to use, but it’s somehow very tricky, especially when you want to check the type for the input. For example, Array in JavaScript sometimes will be treated as special Object, which makes it difficult to check if an input is an array. That’s why you will to use isarray module because once you require the package in your project, you don’t need to worry about that headache anymore. I think using external modules greatly makes developing easier. You don’t need to worry about these small details, instead you could just focus on your core implementation. If you need some helper functions, just require it in your project!

Do you make your own shoes? No, you buy them in a store. Most don’t care how the shoe is made. Just how good it fits.

As node package is usually highly modular, plus the complex dependencies system brought by NPM, it’s common that such small modules are downloaded thousands of time every day.

The real problem

I think this also showed a problem in the node ecosystem: there are no standard common libraries like Apache commons library or Google Guava, which is a single collection of all helpful modules like these one-line modules (loadash is good, but not enough). Developers need these common helper functions, and these helper functions are small modules, and we need to include all these modules in our projects. This Snowball effect will eventually caused the cause that small modules are required by thousands of bigger modules. Once it gets removed or has some bugs, whoever requires it will suffer as well.