Posts by :

Software Coding: an Engineer’s Philosophy

I originally wrote this last year on February 12, 2013. A lot has changed since then, including getting a new job. Yet I am still coming back to the same themes and concepts in my new environment. I feel like I’ve gained enough experience now to post the final draft on this topic, and the existential dilemmas & problems that we face as Software or DevOps Engineers. Even if you’re an Engineer of any kind, you’ll probably relate to these high-level concepts and philosophies, because they really have to do with solving complex problems of any kind in an impermanent world of change.

One day, a co-worker asked me this question:

Q: In the general case: Is it better to simply fix the immediate problem, even if I foresee a possible future problem?  And what do I do if solving the possible future problem creates more problems.  I feel like this question is practically philosophical…

 

It took me a while to think & write up an answer for this, but afterwards I realized that I had some helpful tips to share.

A: Ah, yes…  I am currently also trying to figure out the ideal solution to this philosophical problem Jumping Happy Emoticon

Getting Lost in Problem-Space

Ideally when we encounter a problem, we would want to solve it without creating new ones.  However, in practice there are many cases where solving one problem either creates or reveals a new one (with some sort of cause/effect relationship).  Usually we can happily go along our way fixing each problem as it arises, uncover a new one, and start fixing that.  Hopefully this process leads us towards less problems and some feeling of completion of the task at hand where we can mark it as done.  However, sometimes solving one problem causes a cascade of new problems to arise. Sometimes also we can get lost in the maze of problems and we lose sight of the forest for the trees.

Increase Your Awareness of the Forest

In this case, I have realized that without a full awareness of enough cause/effect possibilities that stem from our decision, it is easy to get lost in the labyrinthine field of “problem-space”.  This can be very scary, discouraging & overwhelming, and give us many mixed feelings and beliefs about which way to go Thinking EmoticonMixed Feelings EmoticonDizzy Frazzled EmoticonQuestioning EmoticonScared Emoticon.  Sometimes this awareness of a potentially infinite minefield of problems can instill the ‘fear of the unknown’ which drives me towards the decision to do nothing, ignore, or procrastinate, but the initial problem still is not solved.  One helpful thing I’ve realized is that we all need rest sometimes, and a break can help to recharge & come back at the problem with full force, yet it’s important to do this consciously rather than self-sabotage or allow it to become procrastination.  We can realize the futility of doing nothing, so rather than “hiding” and withdrawing from engaging in life for too long, I usually end up deciding to do something with as much information as I know in the moment.  Sometimes I end up researching more to find some possible solutions or workarounds and choose the best one, sometimes I break the problem down into pieces and make a small step towards solving the first problem in the moment, and sometimes I ask for help or try to find some expert in the field who can either point me in the right direction or even fix the problem for me. Other times, I’m tempted with implementing a temporary or stop-gap solution. Still other times, I forget my own advice altogether and find myself stuck in a frustrated mood. In these times, it helps to have a friend and coworker to bring my awareness to this state, which helps signal me to take a break, or to pursue another avenue to help me to get unstuck.

Ideal Solutions vs. Quick Hacks & Band-aids

With solutions, generally there is some sort of feeling of confidence as to whether the solution is a quick hack, or a more complete one.  It’s good to trust one’s intuition in this case, but sometimes time prevents us from completing the ideal solution too.  Sometimes I end up searching and debugging down so many levels, running into so many problems and dead ends and poring over so much information that it becomes overwhelming. (Some have come to refer to this as yak shaving).  Over time, we would hope that we could somehow avoid dead ends and quick hacks, and there is some truth to the saying “a stitch in time saves nine”. I tend to prefer seeking the more ideal or elegant solutions in general, however, sometimes we all need some quick hack to get things working. In practice, sometimes we find that due to a deadline, some obstacle, or some other reason, we must implement some sort of quick workaround with hopes to come back and fix it later. The danger in this is that we haven’t solved the real problem, and eventually it can come back around and bite us again (and when enough users run into it, or enough people in the community are blocked by this issue, then a good general solution is important for all!). So again we have the spectrum of beliefs that could either lead us towards a quick hack or towards a more dependable and future-proof solution (although potentially more time consuming). In each case, I’ve come to realize that in the end, it all does come down to a quick decision in the moment of which path to take.  We can realize that the journey of getting there was all part of the solution, but staying in the moment in each moment feels important to me too. In realizing the power of Now, we can regain our balance and choose our path. The context of your present moment contains all the information you need to make the right decision.

Problem Solving Balance

It’s good to stop here and realize that there is a limit to the amount of knowledge and information searching that the human mind & ego can handle before getting tired and simply wanting to settle with any solution, whether or not it’s the best one.  Taking a step away from the problem, “rubber ducking” (ideally with a real person… “getting fresh eyes on the situation”), code reviews, or asking for help can provide what’s needed in the moment to become unstuck.  Hopefully these tools will lead us towards a cascade of solutions and the happy sense of completion I spoke of above Jumping Happy Emoticon.  It’s always a balance, and I’m always learning too.

Some feel that it’s best to avoid getting carried away in a “yak shaving” party, and then use this belief to justify avoiding all potential rabbit holes. However, the trap of avoiding all “yak shaving” hurts us because not all yaks are unnecessary problems to solve (seedefinition 1 here). If I encounter the same problem many times, it becomes a big enough issue that I feel the need to address it fully as its own problem.  In this case, it helps to do a bit more research and map out the problem’s “environment” and the “problem space” a bit in order to find a quicker solution to the problem.  Usually I can solve something and move on without encountering too many “dead-ends”.  However, sometimes I also find myself in a labyrinth of dead ends and getting frustrated with myself.  In this case, usually the problem is that I’ve made myself “snowblind” to the real cause of the problem in the first place.  Again, the methods of breaking it down, stepping back, getting help, or fresh eyes on the issue can help to get unstuck. Finally, In the software world there is also another very promising technique to prevent us from going down too many dead ends while coding: Unit Testing!

Map & Record Your Problem-Space

The easiest way to solve a maze is to put your arm out and follow one wall. You might end up going down a bunch of dead ends, but usually you won’t hit all of them, and you will always find your way out. The easiest way to avoid dead ends is to learn from your past experience and remember which way to go. But what happens when the maze is too big for us to memorize or we don’t remember the way out? Well, you’d probably want to create a map. In software, there is one good way to do this to avoid bug regressions: Automated Testing. The idea behind TDD is to write a test that will verify your code performs the function it should first, then write the actual functional code to get the test to pass. Over time, this idea allows us to create a library of Automated Unit Tests that verify our code works. It also protects us from old bugs recurring, or from introducing a certain set of new bugs as long as our tests are well designed. This method can help improve and streamline our coding cycle by immediately letting us know whether we’ve gone down an old dead end we already knew about, or if we have created a new problem. Essentially we are painting ourselves into the ‘happy path‘ which we will eventually converge upon.

The Philosophical & Existential Despair of Desire Alignment & Problem Solving

Because we are human, we do have desires. As engineers, we tend to be driven to solve problems & have a desire to do so. However, sometimes we don’t truly desire to solve all the problems we are faced with. Sometimes problems are too simple or boring to us, or sometimes we are faced with a multitude of small problems and issues which drag us away from our original problem. Sometimes we just feel so frustrated not knowing “Why things just won’t stay fixed?“. The answer to this question is that this is a world of forms which are constantly shifting and changing. You are not the same person you were 5 years ago, last year, yesterday, and even a moment ago. Software is constantly updating and changing, the applications and operating systems we work on are being developed and improved. Due to the complex dependencies and interconnectedness of these pieces of software, sometimes things end up in a (very) broken state. It’s really a multidimensional shifting puzzle that is constantly evolving over time. Think of some kind of pandimensional hyper-rubiks cube of entangled dependencies. Sometimes problems are too hard due to the number of simple yet interrelated problems. Here’s where workarounds and simplifying things can really help. Sometimes we may decide to give up, or find another way around the problem. Perhaps we may just decide to cut through the gordian knot, and avoid solving the difficult problem altogether. As engineers, and humans with an ego identity, we can tend to see these possibilities as unskillful, or perhaps undesirable. It may feel like giving up, however there is great wisdom in this route. It’s a perfectly valid choice to simplify a problem to the point of neutralizing it altogether. The real hard part here is our own internal struggle with our desires. Alan Watts as always has some wisdom on this topic:

Sometimes I find myself so deep in thought about something that my mind feels stuck in overload of the infinite subtleties of navigating life… yet surprisingly enough I find my way through even the most difficult of circumstances… and often enough I am able to see my thoughts in their “external” parallel thought form expression, as if looking in the mirror. This highlights the imperfections in myself which I fondly look at and love for what they are, in their own time and place. Then I remember that the axiom “as above, so below” holds as well… and it gives me great comfort in knowing that everything is alright, and I’m meant to be doing whatever it is that I’m doing, whether I see it as perfect or not.

“When you make a decision… (people have a great deal of anxiety about making decisions)… So when we decide, we’re always worrying “Did I think this over long enough, did I take enough data into consideration. And if you think it through, you’ll find that you never could take enough data into consideration. The data for a decision in any given situation is infinite. So what you do is you go through the motions of thinking out what you will do about this, and when the time comes to act you make a snap judgement. But we fortunately forget the variables that interfere with this coming outright, it’s amazing how often it works. But worriers are people who think of all the variables beyond their control, and what might happen. So then when you make a decision, and it works out alright, I think very little of it has much to do with your conscious intent and control.” – Alan Watts

Conclusion

At the end of the day, we all usually try our best to come up with good solutions, no matter how difficult or daunting this may be. There is a balance to be found between seeking ideal best-case solutions and implementing quick and usually temporary kludges (keep in mind there are elegant hacks too!). Also, it’s important to note that perfection is an illusion, because perfection is highly subjective (things can always be improved, or be worse). It’s ok to settle for “good enough”, as long as you take an attitude towards continuous improvement. An positive attitude of accepting that mistakes can and will be made (and that’s ok!) with the habit of learning from mistakes creates a direction of evolution towards continuous improvement, while a self-defeating attitude of overwhelm with things being imperfect can lead to an attitude of giving up. Sometimes it is very overwhelming to get lost in a labyrinth of problems. Therefore it can be quite helpful to learn some mental philosophical kung-fu and other techniques which we can use to regain our balance. This is possible without going too far into reviewing the entire spectrum of Agile Software Development philosophy, or too far into software-specific patterns or techniques. The most helpful techniques for Software newcomers are those that can help us feel less overwhelmed and help to re-frame the situation such as: Research, Breaking the Problem Down, Getting Fresh Eyes, Asking for Help, Seeking an Expert, Mapping your Problem Space, Workarounds, Simplification, and Cutting the Gordian Knot. The most helpful technique is to realize the power of choice in every now moment, and mastery is knowing what technique each moment calls for. Over time, we learn lots of techniques and eventually become a master or expert in our field. Every teacher was once a student, and the best teachers are those that learn from their students. I’m always learning more and working on improving too ;-) Happy coding!

Letting Go

Do not allow knowledge to stand in the way of the Truth.

If you want to see the sunset, you have to put aside all the words. Descriptions… the words actually block the experience.

“Too often, the menu is one’s knowledge and the food is the experience.

Lyraphase 007 – Funky Buddha 11-11-11

LyraPhase 007

Funky Buddha 11-11-11

Doesn’t 005 Come next?

Yes, it does normally come after 004…

About it

This second “out of phase” release intends to showcase bits of the deeper and progressive side of EBM that we’ve all come to love. Originally recorded on 11/11/2011 at Funky Buddha lounge, this mix was consciously crafted to start out deep & funky, then gets techy & mixes various genres all intended to make you move with it’s healing harmonic vibrations.

Tracklist:

Here’s the tracklist & the cuesheet

DJ Phasik - LyraPhase 007 - Funky Buddha 11-11-11

01) Joe Dipadova - Consciousness (Main One Mix)
02) Dubfunk - Mandala (Original Mix)
03) Markus Wesen - Tailside (Original Mix)
04) Alex & Filip - Frailty (Original Mix)
05) C-Rock And Bredd Johnson - Bassmatic (Marc Poppcke Remix)
06) Sir K - Buddha Dream (Raditz Room 'Tibetan Mantra' mix)
07) Soliquid - Taipei Pocok Salata (Dub Mix)
08) Soliquid - Princess Sophie
09) Ramon Tapia - Back To The Bush (Original Mix)
10) Loko - Johnny Eres Tu
11) Mikroboy - Vom Leben Und Verstehen (Seelenluft Dub Mix)
12) James Zabiela - Perseverance
Avoid Dependency Issues and Safely Cut Down Your Linux Boot Time With e4rat

Avoid Dependency Issues and Safely Cut Down Your Linux Boot Time With e4rat

If you have been performance tuning your Ubuntu rig you may have run across a nifty tool called e4rat. This tool allows you to optimize the location of files your system accesses during its boot process, and pre-load them in memory in order to speed up boot times. This idea is definitely not new, as there have been similar tools to do this in the past such as ureadahead or sreadahead. However, e4rat is a new tool which uses some recent Linux kernel functionality introduced alongside the ext4 filesystem. On the e4rat homepage, the author goes into more detail:

e4rat (“Ext4 – Reducing Access Times”) is a toolset to accelerate the boot process as well as application startups. Through physical file realloction e4rat eliminates both seek times and rotational delays. This leads to a high disk transfer rate.
Placing files on disk in a sequentially ordered way allows to efficiently read-ahead files in parallel to the program startup. The combination of sequentially reading and a high cache hit rate may reduce the boot time by a factor of three, as the example below shows.

e4rat is based on the online defragmentation ioctl EXT4_IOC_MOVE_EXT from the Ext4 filesystem, which was introduced in Linux Kernel 2.6.31. Other filesystem types and/or earlier versions of extended filesystems are not supported.

 

This tool may be for you if you:

  • Use Linux
  • Use ext4 filesystem
  • Have a non-SSD Hard Drive
  • Want to speed up your boot time!

However, there is a rather unfortunate dependency issue that might have bugged some people (including myself). Currently in Ubuntu 12.04 e4rat conflicts with ureadahead, while ureadahead is a part of the ubuntu-minimal metapackage[1]!

This caused the problem of having to uninstall ubuntu-minimal if you wanted to use the e4rat package, potentially opening the door to unexpected problems for your next dist-upgrade. On top of this, e4rat requires that you modify your init kernel option in grub.cfg, and may not clean itself up when you remove the package. This could potentially leave your system in a non-bootable state if e4rat is removed until you fix your grub config.

To address this problem and make things safer, I’ve created a PPA containing a rebuilt ubuntu-minimal package without the ureadahead dependency. It also includes a rebuilt e4rat package containing a postrm cleanup script to remove any init=e4rat-* kernel options you may have added to your /etc/default/grub file.

Here’s what will happen:

  1. Upon dist-upgrade, this PPA will be disabled in /etc/apt/sources.list.d/trinitronx-minimal-no-ureadahead-precise.list.distUpgrade
  2. The dist-upgrade will continue because this ubuntu-minimal package is still installed
  3. ubuntu-minimal will be upgraded to the latest version, *with* dependency of ureadahead
  4. apt will detect a conflict with e4rat, and remove it when installing the new ureadahead package
  5. If you have my e4rat package installed, the postrm script will take care of your /etc/default/grub entry, and re-run update-grub to update your /boot/grub/grub.cfg
  6. Your system should now reboot fine!

How to install & use e4rat

To install e4rat from my PPA (and avoid potential issues later down the line), follow these simple steps:

Install my PPA, Remove stock ubuntu-minimal and ureadahead & install re-packaged ubuntu-minimal & e4rat

    sudo apt-add-repository ppa:trinitronx/minimal-no-ureadahead
    sudo dpkg --purge ureadahead ubuntu-minimal
    sudo apt-get update
    sudo apt-get install ubuntu-minimal e4rat

Setup e4rat to Optimize Your Boot Time

After you’ve got it installed, you’ll follow the same steps helpfully outlined in this article.

1. Collect

First e4rat needs to analyze the files accessed within the first 120 seconds of your system boot process. If you wish to override the default timeout of 120 seconds, you’ll need to edit your /etc/e4rat.conf before you reboot, and uncomment the line that looks like this:

; timeout 120

And change it to however long in seconds you want. Here’s an example for 4 minutes:

timeout 240

To collect the list of files, you’ll need to reboot and tell grub to run e4rat-collect first thing (the init process is the first process started when booting). To do this, follow these steps:

  1. Reboot your system & wait for the GRUB menu to pop up. When it does, highlight your Linux kernel and hit the “e” key to edit the configuration. Search for the kernel line, which varies a bit on everyone’s system, but should look something like this:
    linux   /boot/vmlinuz-3.2.0-36-generic root=UUID=badcafe0-f00d-babe-feed-facef00d0000 ro   quiet splash
  2. Add init=/sbin/e4rat-collect to the end like so:
    linux   /boot/vmlinuz-3.2.0-36-generic root=UUID=badcafe0-f00d-babe-feed-facef00d0000 ro   quiet splash init=/sbin/e4rat-collect
  3. Hit Ctrl+X or F10 to continue booting. e4rat will collect a list of files accessed within the first 120 seconds (or whatever timeout you chose)
  4. After 2 minutes (or your timeout) is up, check that /var/lib/d4rat/startup.log exists.
    ls -l /var/lib/e4rat/startup.log 
    -rw-r--r-- 1 root root 307549 Nov 26 11:50 /var/lib/e4rat/startup.log

    If so, you can proceed. (Feel free to check what files it found too ;-)

Note: After rebooting for this step, you may want to follow this helpful tip from LifeHacker:

For the purposes of collecting the right files, act as you normally would when you boot. E.g., if the first thing you do when you boot up your computer is open your browser and mail client, do that after you finish booting here. It’ll move those files too so those programs launch extra quickly. Once your two minutes are up, you can stop “acting”.

 

2. Reallocate

Next, reboot your computer again and edit your Grub line as before. This time, add the word single to the end of the kernel line like so:

linux   /boot/vmlinuz-3.2.0-36-generic root=UUID=badcafe0-f00d-babe-feed-facef00d0000 ro   quiet splash single

This should reboot you into a terminal. (if you get a blue screen, pick the option to boot as normal) You can log in using your username and password. Once there, execute this command:

sudo e4rat-realloc /var/lib/e4rat/startup.log

3. Preload

Now you just need to set e4rat-preload to run first thing on boot. This time to make the change persistent, we will edit the /etc/default/grub config file, and change the GRUB_CMDLINE_LINUX_DEFAULT boot line. First open your favorite editor (feel free to use vi instead of nano if you wish).

sudo nano /etc/default/grub

Add “init=/sbin/e4rat-preload” to the end of the variable like so:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash init=/sbin/e4rat-preload"

Save the file and close it. Then run:

sudo update-grub

Congratulations, you’re done!  On your next reboot, you’ll enjoy a much faster boot time!


DISCLAIMER: Although I have unit-tested the various pieces that will happen during the dist-upgrade process, I do not guarantee that you won’t run into any problems, so if you are in doubt, are paranoid, or are just extremely unlucky please remove e4rat & check your /etc/default/grub and /boot/grub/grub.cfg manually.

If something goes terribly wrong, you’ll probably need to remove the “init=/sbin/e4rat-preload” grub parameter from your /etc/default/grub file and run sudo update-grub manually.

If you are stuck with a kernel panic on booting, just reboot, press ‘e‘ to edit your grub command line, and remove any ‘init=/sbin/e4rat-*‘ you see there, then press F10 to boot.

[1]: https://help.ubuntu.com/community/MetaPackages

Seeking non-Duality

Seeking non-Duality

In awareness of the problems facing our planet at this time, as well as the intellectual and internal premise upon which these most likely arise from, I have felt the urge to stress the importance of finding inner peace amongst the turmoil.  Bashar has said “The degree of which we are experiencing any darkness is the degree to which we will also experience the light”[1].  So no matter what circumstances happen to be arising, remember that this too shall pass.

Bashar's Rubber Band Analogy

[1]: Like a Rubber Band all of the separated and charged energy in our world is ready to take us in a new direction. For every action, there is an equal and opposite reaction.

In the spirit of a love based non-duality principle, let us create the template of a more loving global community.  One in which we collectively choose to foster peace, compassion, understanding, love, sharing, happiness and joy.  We currently have all the tools and technology to solve our problems, now we just need to share, collaborate and seek solutions rather than more problems.  If humanity were to apply the collaborative spirit of the Open Source movement to other areas, many of the blockages which caused the current problems could be cleared away. We have heard it from many spiritual masters and humanitarians of the past: Peace starts from within

“We can never obtain peace in the outer world until we make peace with ourselves.” – Dalai Lama

“Non violence begins with learning how to be less violent and more compassionate towards ourselves. We learn by building the courage to speak and act with respect. honor and reverence for our own being.” – Ghandi

“It’s better to light a candle than curse the darkness.” – Peter Benenson

The people have the conscious choice within to choose peace in ANY moment. I choose peace NOW, and will try my best to choose it in every moment. I hope you will join me ^_^

In a dualistic paradigm, peace begins with awareness of the problem & forgiveness.

“People can expand their mental boxes and learn to see beyond outer differences, including race, ethnicity, nationality and sex. They can develop a true community spirit as human beings – or even better, as spiritual beings – and transcend the differences that normally divide people. In its essence, the purpose of the experiment is to break the isolationism that most groups of people have and force people to become more open to the universality behind appearances (Judge not according to the appearance, but judge righteous judgment. (John 7:24)). The ultimate universality being, of course, that all people are the offspring of the same God.” [2]

 What exactly is Non-Duality?

Alan Watts is able to expound upon this concept:

” How would you feel if you saw everything as really one basic reality? Well, a lot of people think that it would be as if all the outlines and differentiations in the field of vision suddenly became vague and melted away and we saw only a kind of luminous sea of light.
However, rather advisedly, the Vedanta philosophy does not seriously use the word ‘one’ of the supreme self because the word and idea ‘one’ has its opposite ‘many’ on one side, and another opposite, ‘none’, on the other. It is fundamental to Vedanta that the supreme self is neither one nor many, but as they say, non-dual, and they express that in this word ‘advita’. A is a negative word like non. Dvita is from dva, same as the Latin duo, two. So advita is non-dual. At first this is a difficult conception because naturally, a Western logician would say, ‘But the non-dual is the opposite of the dual. Therefore, it has an opposite.’ This is true, but the Hindu is using this term in a special sense. On a flat surface I have only two dimensions in which to operate so that everything drawn in two dimensions has only two dimensions. How, therefore, on a two-dimensional level, can I draw in three dimensions? How, in logic, is it humanly rational to think in terms of a unity of opposites? All rational discourse is talk about the classification of experiences, of sensations, of notions, and the nature of a class is that it is a box. If a box has an inside, it has to have an outside. ‘Is you is or is you ain’t?’ is fundamental to all classifications, and we cannot get out of it. We cannot talk about a class of all classes and make any sense of it. However, on this two-dimensional level, we can create, by using a convention of perspective, the understanding of a third dimension. If I draw a cube, you are trained to see it in three dimensions, but it is still in two. However, we have the understanding that the slanting lines are going out through the back to another square, which is behind the first one, even though we are still on two dimensions. The Hindu understands this term advita as distinct from the term ‘one’ to refer to that dimension. So when you use the word advita, you are speaking about something beyond duality, as when you use those slanting lines you are understood to be indicating a third dimension which cannot really be reproduced on a two-dimensional surface. That is the trick. It is almost as if whatever we see to be different is an explicit difference on the surface covering an implicit unity. Only it is very difficult to talk about what it is that unifies black and white. (Of course, in a way the eyes do. Sound and silence are unified by the ears). If you cannot have one without the other, it is like the north and south poles of a magnet. You cannot have a one-pole magnet. True, the poles are quite different; one is north and the other is south, but it is all one magnet. This is what the Hindu is moving into when he is speaking of the real basis or ground of the universe as being non-dual. Take, for example, the fundamental opposition that I suppose all of us feel, between self and other – I and thou – I and it. There is something that is me; there is an area of my experience that I call myself. And there is another area of my experience which I call not myself. But you will immediately see that neither one could be realized without the other. You would not know what you meant by self unless you experience something other than self. You would not know what you meant by other unless you understood self. They go together. They arise at the same time. You do not have first self and then other, or first other and then self; they come together. And this shows the sneaky conspiracy underneath the two, like the magnet between the two different poles. So it is more or less that sort of what-is-not-classifiable (that which lies between all classes). The class of elephants opposite the class of non-elephants has, as it were, the walls of the box joining the two together, just, as your skin is an osmotic membrane that joins you to the external world by virtue of all the tubes in it, and the nerve ends, and the way in which the external energies flow through your skin into your insides and vice versa.” ~Alan Watts

So we are left with this idea of something which lies between all classes that unifies the opposites. I believe this lies in some form of understanding of spiritual Truth, which is beyond yet inclusive of all classes and all opposites. When we try to apply this non-dual perspective of that which is in a higher dimension to our experiences here on earth, things might get a bit confusing, especially without a tangible experience of this presumed higher dimension. I believe that in order to experience this higher dimension, some form of spiritual path must be followed. So if we are to say that all of our global problems arise from a level of consciousness in which separation-from and fear-of one another are the predominant ideas, then logically the solution would be perhaps to seek a non-dual form of consciousness. I believe that this form of consciousness is one based upon unconditional love for both self and one another. Many spiritual traditions and paths (at their True core) seem to be aligned with this and similar love-based qualities. However, this decision and choice between love and fear lies in each moment, and sometimes we can get caught in the collective fear-based separation consciousness, and make mistakes that may hurt one another. This can cause a meandering of the path, a seeking or more journeying in order to find the solution. So we must also seek qualities of forgiveness, compassion, tolerance, and acceptance towards one another, regardless of any conditions or qualities that we might project upon ourselves, another person, or group. From our perspective of each being living in a society in which parts may seem at odds, for some it seems that an extreme force and a government controlled paradigm are attractive. For others to which free will is an ideal, a more libertarian or “Laissez-faire” perspective is attractive. However in reality, a balance, or middle way between the two extremes is probably the best for the collective.

I believe that no matter the circumstances, it is always best to choose love. This can mean many things depending on each circumstance or moment. All beings have some level of free will, and so showing love amongst members of society cannot be “enforced”, and enforcing love would not be true love now would it?

“Love cannot be controlled. It cannot be predicted. It cannot be enforced. It cannot be enslaved. It cannot be killed. It cannot be defined. If you would speak of the depth, the power, and the infinite genius of love, think upon its ability to end all duality. Love has no opposites. It cannot be captured, and yet it willingly serves. It is infinitely free, and yet it abides in captivity. It cannot be enforced, and yet it is the basis of all law. It is the bringer of truth, and yet it never judges. It is the seer of all things, and yet it never condemns. It conquers by surrendering. It rules by lifting up. Through its mastery of paradox, love ends all duality.” – from Love Without End by Glenda Green

Rather than seeking to control the tools which people could use as weapons, rather than giving up our own personal freedoms, rather than giving into the collective unconscious fears and polarizing ideas brought up by any incident… instead let there be a space for love instilled in the hearts of every man and woman. For this alone can prevent all violence. Let those trapped in fear be awakened into the Light of the Love within their own hearts, and the hearts of all humanity. And let this principle of the most fundamental human emotion reflect both inwards and outwards, spreading a shockwave of compassion, forgiveness, understanding, peace, joy, happiness, and trust in God and our fellow humanity. All men and women are created equal, and with equal responsibility to themselves and one another to protect and reflect the innermost flame of Life itself.

How can we reflect you may ask?  By first seeking to enlighten oneself, then sharing the knowledge we find with others. This at times can be a grueling inner-struggle. We may find that we are repressing something or that we are holding some belief that is not serving our highest good.

Can a blind man lead a blind man? So here’s my question with the occupy movements etc. or any movements for change…which target an opposition outside of themselves. If i throw the money changers out of the external temple but they are still in the internal temple (the mind of the one doing the throwing out)? What occupy movement will take care of the money changers in the inner temple? This is why the true masters showed the lonely way of inner change…first. Only real inner work on ur own shadow (inner opposition) and finding inner solid ground (Self) will immunize u from being hijacked into fake staged external left – right conflict, and see outside the illusion what is really going on. As long as we remain divided within we will be hijackable into fake dichotomies without. These fake dichotomies are not only political leftist versus rightist seeming contradictions, but almost everybody u meet…talk to, every talk u hear and give, everyone u ask for advice or try to help, every program u follow…every thing u encounter on a daily basis…will try to recruit u into some form or the other of “either or”, we versus them, black versus white….thinking…feeling. Every next trap u steer free from being seduced into taking sides…u have conquered the great force of Maya: the great illusion of duality. The truth behind the veil is that the poles r secretly one…This is why we so need the opposite to be One. The real program behind all left-right programs is to keep u in Maya (The Matrix)…or in division. They say fear is the ultimate control mechanism. It is not the ultimate one, division is. Yet beware when the manufactured division (chaos) is escalated toward its Zenith. For then a false Order (fake union) will be heralded in as salvation (ultimate ‘Solution’). Beware when they SAY Peace and Security…This will not be “real”, nor can it be a Hegelian solution to manufactured dissent, i.e., the full destruction of the opposition (repression of the shadow). The solar hero resulting from this full excision of the opposite will be raptured down over the wrong rainbow. The real one is a result of the integration of the opposites, not of renting them apart. Ojo!” -Dimitri Halley

To do this, we find that we have to be able to surrender to something higher than our little self.  One has to take the beam out of our own eye and allow oneself to see nakedly, and have the great courage to present ourselves nakedly to others. This can be a difficult process.  We might not always like the picture in the mirror, but in order to change it we don’t try and paint over the mirror.  Instead, one would seek to change something in oneself to get the reflection to change.

Naked Surrender

Ok, I’m not going to lie. Sometimes this work is *really* hard. We might be faced with a set of terrible fears or circumstances beyond our control. Facing one’s own shadow is difficult. Dmitri Halley has more to say about this:

The shadow work is becoming naked before the light…nothing more to hide from myself. We seek to cover up our feelings of depression and anxiety; inadequacy, insecurity, abandonment, lack of foundation, never been backed up, emptiness…with all sort of highs/attachments (masks/coverings/cover stories); money, control, attention, power, food, plans… Yet God wants us to be naked. To experience life completely sober…no matter where it takes us or precisely because of where it takes us. To dare to accept the insecurity without covering it up with seeking assurance…To accept these feelings raw, honestly and within this contained vulnerability to let go/d (Self). Naked like birds who don’t plan for the next meal. Miracle cannot show up in ur life if u are driven by plans, insurances and huge salaries. The universe cannot conspire in ur life if u r not naked. The Nazarene appeared in a dream to a woman waiting for economic stability (a deal) before surrendering to Self and wherever “Thine Will” wanted to take her. He said: “Do u not remember I turned water into wine and stones into food.” The unadulterated light cannot shine in our sea of pisces if we keep covering up our shadow. -Dimitri Halley)

Resistance to this difficult work might have something to do with the current state of the world, and the course of human history. Alan Watts gives us many insights about the psychological process of recognizing that the potentials for both good and evil lie as much in oneself as in the other.

King & Queen on chessboard

Alan Watts – Seeing Through the Game (Carl Jung Tribute)

About .CRX Files

About .CRX Files

Recently, while trying to install the cr-gpg Google Chrome Extension (blog post to come), I ran into a small problem trying to import the .crx file. This led me to find out what the .crx file type is, and how to extract it.  As most of the posts on this blog are for rather advanced linux users, I’m going to try and make this more general and helpful for the general public.

TL;DR

The short answer: unzip!

It’s a .zip file with an extra header containing the author’s public key and GPG signature.  You may want to strip it if you’re repackaging it.

The file specification can be found on Google’s .crx Package Format page.

The Long Answer:

Question: What is a CRX File?

Did you find a CRX file in your web browser’s “Downloads” folder on your computer and wonder what program should open it? Maybe you’ve been trying to install an unofficial Google Chrome Extension and got the message “Apps, extensions, and user scripts cannot be added from this website” when opening the .crx file.

Answer:

A file with the CRX file extension is a Google Chrome Browser Extension Archive file.

According to a quick google search: CRX files might also be “Links Games Course” files (Although it comes up as a top result in Google, I couldn’t find any other info on this… I’m unsure if these are even a valid file type, or just google search spam?).

How To Open an CRX File:

As you probably know, the easiest way to open any file is to double-click it and let your PC decide which default application should open the file. If no program opens the CRX file then you probably don’t have an application (ie: Google Chrome) installed that can view and/or edit CRX files.

Warning: If you are on Windows, beware when opening executable file formats received via email or downloaded from websites which you are not familiar with. See this List of Executable File Extensions for file extensions to avoid and why.  (If you are on linux, be smart, be secure, and be happy! ^_^)

The CRX file type is primarily associated with the ‘Google Chrome’ web browser by Google. Any file with extension CRX is likely a plugin file or as it is more commonly known: a Google Chrome Extension file.  These files are used to package a Google Chrome extension, and can be installed in Google Chrome to add extra features to the browser.

The Google Chrome browser uses CRX files to provide extend-ability in the browser program. A Google Chrome CRX file is really just a renamed ZIP file with an extra bunch of bytes in the header to verify the plugin’s origin (and validate the private key’s signature using the public key).  This is all done for security purposes, as we would not want to run or install any browser plugin from a source that we do not trust.

In theory, any archive/compression program, like 7-Zip, TUGZip, unzip, MacZip (all free) OR Winzip/WinRAR (non-free), will open CRX files for extraction (expansion/decompression).  CRX files may also be opened using any other archive tools you may be familiar with to view the contents of the packaged plugin/extension.  However, depending on whether your tool can ignore the file header correctly, you may need to strip it first to convert to a zip, or use another tool.

If you find that an application on your computer does try to open the .CRX file but it’s the wrong program or if you would rather have another installed program open CRX files, you might need to Change the Default Program for this file extension.  (If you’re on a Mac, please see this 4 step tutorial on how to change the default application for a filetype).

How To Convert a CRX File to a ZIP file:

As of this writing, there is no way to open the CRX file in its default program (Google Chrome) and choose to save the open file as another file format.  However, you may want to try 7-zip to extract it, modify to your liking, and then repackage it as a .ZIP.

There is one basic way to attempt to convert a CRX file to a ZIP file:  strip the extra header!

Important: You cannot usually change a file extension (like the CRX file extension) to one that your computer recognizes and expect the newly renamed file to be usable. An actual file format conversion using one of the methods described above must take place in most cases.

To do this job, we’re going to have to resort to some basic unix commands: dd and tail

To strip the header, you’ll need to know how long it is first. Luckily, InfoZIP‘s unzip utility can tell us how long it is (tested on Ubuntu and CentOS with unzip versions 6.00 and 5.52 respectively):

# Get unzip if we don't have it
[ -z "$(which unzip)" -a -n "$(which apt-get)" ] && sudo apt-get -y install unzip
[ -z "$(which unzip)" -a -n "$(which yum)" ] && sudo yum -y install unzip
unzip -l file.crx 

Sample output:



Archive:  file.crx
warning [file.crx]:  306 extra bytes at beginning or within zipfile
  (attempting to process anyway)
  Length      Date    Time    Name
---------  ---------- -----   ----
      175  1980-00-00 00:00   background.html
     5047  1980-00-00 00:00   background.js
        0  1980-00-00 00:00   binaries/
    19860  1980-00-00 00:00   content_script.js
       58  1980-00-00 00:00   custom.css
     4224  1980-00-00 00:00   general.html
     4400  1980-00-00 00:00   icon.png
        0  1980-00-00 00:00   images/
    93868  1980-00-00 00:00   jquery-1.7.1.min.js
    32709  1980-00-00 00:00   jquery-ui-1.8.11.thinkst.css
   370776  1980-00-00 00:00   jquery-ui-1.8.13.custom.js
    11674  1980-00-00 00:00   jquery-ui-1.8.13.custom.min.js
      811  1980-00-00 00:00   jquery-ui-content.hack.js
     2736  1980-00-00 00:00   jquery.client.js
    11688  1980-00-00 00:00   jquery.ui.widget.js
     1273  1980-00-00 00:00   manifest.json
     2523  1980-00-00 00:00   options.html
     2206  1980-00-00 00:00   options.js
     4462  1980-00-00 00:00   popup.html
     2050  1980-00-00 00:00   popup.js
     1776  1980-00-00 00:00   images/sprite_black2.png
     1633  1980-00-00 00:00   images/verify.png
        0  1980-00-00 00:00   binaries/Mac/
        0  1980-00-00 00:00   binaries/Ubuntu/
        0  1980-00-00 00:00   binaries/Windows/
    45568  1980-00-00 00:00   binaries/Windows/libgpg-error-0.dll
   235520  1980-00-00 00:00   binaries/Windows/libgpgme-11.dll
  1027584  1980-00-00 00:00   binaries/Windows/npwebpgPlugin-v0.5.9.dll
  2305852  1980-00-00 00:00   binaries/Ubuntu/npwebpgPlugin-v0.5.9_32.so
  1768658  1980-00-00 00:00   binaries/Ubuntu/npwebpgPlugin-v0.5.9_64.so
        0  1980-00-00 00:00   binaries/Mac/npwebpgPlugin-v0.5.9.plugin/
        0  1980-00-00 00:00   binaries/Mac/npwebpgPlugin-v0.5.9.plugin/Contents/
     2082  1980-00-00 00:00   binaries/Mac/npwebpgPlugin-v0.5.9.plugin/Contents/Info.plist
        0  1980-00-00 00:00   binaries/Mac/npwebpgPlugin-v0.5.9.plugin/Contents/MacOS/
        0  1980-00-00 00:00   binaries/Mac/npwebpgPlugin-v0.5.9.plugin/Contents/Resources/
        0  1980-00-00 00:00   binaries/Mac/npwebpgPlugin-v0.5.9.plugin/Contents/Resources/English.lproj/
      150  1980-00-00 00:00   binaries/Mac/npwebpgPlugin-v0.5.9.plugin/Contents/Resources/English.lproj/InfoPlist.strings
      411  1980-00-00 00:00   binaries/Mac/npwebpgPlugin-v0.5.9.plugin/Contents/Resources/English.lproj/Localized.rsrc
  3462932  1980-00-00 00:00   binaries/Mac/npwebpgPlugin-v0.5.9.plugin/Contents/MacOS/webpgPlugin
---------                     -------
  9422706                     39 files

You should see the number of extra bytes output as:

warning [file.crx]:  NNN extra bytes at beginning or within zipfile

Now all we need is to strip the extra bytes. There are two main ways to attempt to strip the extra header (and convert) the CRX file:

  1. Use dd (set skip=NNN bytes)
    dd if=cr-gpg-0.8.3.crx of=/tmp/file.zip bs=1 skip=306
  2. Use tail (use -c +NNN+1 bytes. Ex: NNN=306 => use 307)
    tail -c  +NNN+1 file.crx > /tmp/file.zip
Installing latest git on Ubuntu with git-subtree support

Installing latest git on Ubuntu with git-subtree support

Recently I found it necessary to join two git repos together while still maintaining history & future ability to split or rejoin them.  Enter the ‘git subtree‘ command.

Because this command is now merged into git-core since 1.7.11, we will need to install the latest git from a PPA.  Note that as of ‘now‘, the latest available package from the PPA we will use is 1.8.0, and it currently installs the subtree script to /usr/share/doc/git/contrib/subtree. However, since the Makefile there expects asciidoc.conf to be in ../../Documentation/asciidoc.conf, we must checkout the source package & make from there.

I am using Ubuntu 12.04.1 LTS here.

Installing latest git-core + git-subtree

First add the PPA, update & upgrade.  Then install git packages that are held back by apt.  Also install asciidoc (optional if you want the manpage).

sudo add-apt-repository ppa:git-core/ppa && sudo apt-get update && sudo apt-get -y upgrade && sudo apt-get -y install git git-man git-svn asciidoc

Next, install the source package & make git-subtree + manpage:

[ ! -e '~/src/git-core' ] && mkdir -p ~/src/git-core
cd ~/src/git-core && apt-get source git-core
cd ~/src/git-core/git-*/contrib/subtree/
[ -e '/usr/lib/git-core' ] && sed -i -e '/^libexecdir.*/ s|/libexec/|/lib/|' Makefile || echo '/usr/lib/git-core does not exist! Check that your libexec dir exists and reinstall git-subtree'
sudo make prefix=/usr && sudo make prefix=/usr install && sudo make prefix=/usr install-doc

This may not work for you if you’re not using Ubuntu (your prefix or libexec dir may be different).  If in doubt, get git-core from upstream and build it from there, or install the script to $(git --exec-path)/git-subtree and chmod 755 it (see Makefile & INSTALL doc in contrib/subtree).

Now you should be able to use the ‘git subtree‘ command.  For help, run ‘git help subtree‘ or ‘man git-subtree‘.

Some helpful examples of git subtree in use in the wild:

 

 

Update 2012-10-22:  Currently the package for git 1.8.0 from the PPA does not include the git-prompt.sh script.  If you are using the __git_ps1 function in your bash prompt, you’ll need to get the source package and find that file in /home/jcuzella/src/git-core/git-1.8.0/contrib/completion/git-prompt.sh. Install that file somewhere under your home directory and then add a line to your .bashrc file to source it. You’ll know if you need it because you’ll probably see this message after installing latest git:

__git_ps1: command not found
[ ! -e '~/src/git-core' ] && mkdir -p ~/src/git-core && cd ~/src/git-core && apt-get source git-core
[ ! -e '~/src/bin' ] && mkdir ~/bin
cp ~/src/git-core/git-*/contrib/completion/git-prompt.sh ~/bin/
chmod +x ~/bin/git-prompt.sh
echo '. $HOME/bin/git-prompt.sh' >> ~/.bashrc

Potential Drupal XSS flaw found

On November 22nd, I discovered two vulnerabilities in sites based on Drupal Core 7.9 with default configuration.  These were:

  • an automatic remote phishing vulnerability (automated email sent from drupal user’s website can contain links to an attacker’s site!)
    Suggested CVSS v2.0: AV:N/AC:M/Au:N/C:P/I:P/A:N/E:POC/RL:U/RC:C (What’s that?)
    Suggested Drupal Security Risk Level: Moderately Critical (3 of 5)
  • a potential XSS vulnerability (High Access Complexity… attacker must have MITM or control of a Proxy)
    Suggested CVSS v2.0: AV:A/AC:H/Au:N/C:P/I:P/A:N/E:POC/RL:U/RC:C
    Suggested Drupal Security Risk Level: Less Critical (2 of 5)

The technical details of this vulnerability have been removed until further notice from the Drupal security team ;-)

New Project: Spyder

I decided to write a simple web spider in order to learn Python, and to generate a list of urls for webserver benchmarking & stress testing… and so Spyder was born. It is written in Python 3.

Github:

https://github.com/trinitronx/Spyder

Spyder:

a simple web spider written in Python 3

When called on a url, it will spider the pages and any links found up to the depth specified.
After it's done, it will print a list of resources that it found.
Currently, the resources it tries to find are:

images   -  any images found on the page (ie: <img src="THIS"/>)
styles   -  any external stylesheets found on the page.  CSS included via '@import' is currently only supported if within a style tag!
(ie: <link rel="stylesheet" src="THIS"/>  OR <style>@import url('THIS');</style> )
scripts  -  any external scripts found in the page (ie: <script src="THIS"> )
links    -  any urls found on the page.  'Fragments' are discarded. (ie: <a href="THIS#this-is-a-fragment"> )
emails   -  any email addresses found on the page (ie: <a href="mailto:THIS"> )

An example script for doing something like this, 'www-benchmark.py', is included.  It uses apache benchmark as an example.
Eventually I'll be experimenting with 'siege' for benchmarking & server stress-testing.

NOTE: Currently the spider can throw exceptions in certain cases (mainly character encoding stuff, but there are probably other bugs too)
Getting *working* character encoding detection is a goal, and is sorta-working... ish?  Help in this area would be appreciated!
Filtering the results by domain is almost working too

My Photoshop Artwork

Recently, I’ve received a couple requests to use some of my Photoshop artwork for various purposes.  Up to this point, I’ve released all of my large pieces of art under a Creative Commons Attribution Share-Alike 3.0 Unported license.  This means you can feel free to use them and create new works based upon them as long as you give me credit somehow.  Feel free to contact me, however, because I’d love to know that other people are using it ^_^

So, as long as you mention that I’m the creator of this work somehow, then it’s all good ;-)

In case it’s not clear, the creative commons site makes this a little less confusing.  Just click on the following image:

Creative Commons License

Get Adobe Flash player