In the rabbit hole

Don’t be lazy

2023-06-30T00:00:00+02:00

I consider lazyness a quality for developers. Let me explain.

If I do something twice, there is a good chance I’m going to need to do it a third time. Therefore, I want to help my future self to not do it a third time, to be lazy by avoiding future work.

This precept can take different flavors. For example, I can factorize a common functionality into a function, or into a script. Or maybe this is something that can be automated: doing a release, deploying, testing.

But if you take a close look, it’s not exactly lazyness, because you need to engage more work now, to save some work later. It’s a tradeoff. And like with any tradeoff, you might first want to evaluate:

how much time will it take to factorize or automate it?
how probable the situation is going to happen in the future?

Let’s make a truth table out of it.

#	Time to Factorize/Automate	Probability of happening again	Spend the time?
1	Long	Low	No
2	Short	High	Yes
3	Long	High	🤷
4	Short	Low	🤷

Time to Factorize/Automate

Probability of happening again

Spend the time?

Long

Low

Short

High

Yes

Long

High

🤷

Short

Low

🤷

And today, I’d like to talk about something clearly in the category 2 and yet, too often, people don’t spend the time.

Let’s take an example to illustrate. Let’s say I’m writing a script. At some point in the script, there is the following sed command.^[1].

sed -niE -l 80 's/Referer/Referrer/g' request.http

What do you think -niE and -l does? You don’t know? Well this is a problem because you’re not the only one who is going to read that piece of code. Since there is much more people who are going to read that code, than to write it, making it readable should be a priority over conciseness.

Now, how about that version?

sed --silent --in-place --regexp-extend --line-length=80 's/Referer/Referrer/g' request.http

It’s much better. Not perfect, but much better. You’re going to spend much less time (or no time?) in the documentation trying to understand what each of the option means.

So, dear developers, please, pretty please, in scripts, always choose the long version for options. It’s going to save a lot of time to the future readers of the code, including you. 😉

1. Wondering about this typo? Take a look here.

Make it look like rainbows 🌈

2022-02-22T00:00:00+01:00

This post is part of an ongoing series about tests in Rust:

What can we do to improve the reporting?

In the previous post, we ended up with the following report when running tests.

stderr:
thread 'main' panicked at '
runit-assertions/src/lib.rs:66
    ✓ should be not equal to `0`, was `42`
    ⨯ should be not equal to `42`, was `42`
'

This is already nice but can we improve on that? First of all, it be real nice to color it. How about we use red/green colors to quickly identify which tests failed?

Colors in the terminal

To display colors in the terminal, I searched for a crate with the keyword color in crates.io (see link). ^[1]

A few of them showed up and all are done by known people ^[2] in the community:

ansi_term by Josh Triplett (owner on cargo) and Benjamin Sago (author of exa)
termcolor by Andrew Gallant also know as BurntSushi (prolific author of at least regex, csv, ripgrep to name only a few)
term by Alex Crichton (owner on cargo), Huon Wilson and Steven Allen
console by Armin Ronacher (which I’ve known through cargo-insta) and Pavan Kumar Sunkara

Ok, that is a lot of dropped names but that is also a way for me to thank them. I’ve known by name most of them, interacted with some of them. But for sure, all of them brought a lot to the Rust ecosystem: thank you.

Now, which one to choose? To be honest, I’ve picked the most downloaded one: ansi_term. It’s not the most recently updated (ansi_term: 2 years, termcolor: 1 year, term: 1 year, console: 4 months) but it is the most downloaded one by about 15 millions. In any case, it should not be to difficult to change for another one if that was necessary so I’m not spending too much time thinking about it.

Additional note, most of these libraries basically brings colors but also bold, underlined and a few other stylish stuff.

Let’s make it green and red

First of all, I’m not gonna get through code here because it’s not really interesting. The only technical part worth mentioning is that I’ve made ansi_term an optional dependency, although part of the default features. This means that I want colored output to be part of the default provided behavior of runit… but people should be able to suppress that if they believe it’s just fancy and add too much to the building time (even if it’s probably going to be test building).

First iteration

Very simple start. How about this?

Some colored checkmarks or crosses with a flashy green and red color. On top of that, some bold fonts for the location. Pretty good already, right? We can do more.

Second iteration

How about we highlight more the failed tests and make the passed tests a bit more discrete.

Now, the entire background’s line is colored. The failed tests are more flashy with bold, whereas the passed tests use a more pastel color with normal font.

Can we do more? Let’s try!

Third iteration

The inspiration for this third iteration is purely inherited from pretty_assertions.

First of all, compared values are aligned to ease the comparison. I also used a diffing algorithm thanks to dissimilar to precisely show the difference between the strings. I you’re interested in the details of how complex that nice feature was to implement, it’s only about 27 lines of code. dissimilar is really pulling all the hard work for us.

Since the output is now on 2 lines, I’ve underlined some lines to create a separation between each assertion.

I like this last rendering… although, there is a bunch of problems with it.

Problems to address later

When I worked on these features, I worked head-on on the coloring improvement. Now that it’s done, I can take a step back and think a bit. So far, I’ve been exploring the rendering of only is_equal_to (and its friend is_not_equal_to). But the plan always was to expand to more assertions.

What good is going to be a diff output for something like is_greater_than? Yeah, you’re right, it doesn’t make any sense.

How about if we start using regular expression and have a matches assertion?

Or asserting on a map with has_key or contains?

And I’m sure I could find thousands of these little examples where a diff would not make any sense. It feel like a diff is only useful when comparing for equality which is exactly what PartialEq is about.

Let’s pin these and we’ll try to address them on a later post. One thing that seems to emerge is that the reporting is tightly coupled with the assertion. In term of code architecture, it probably means that the current version, that you can find on woshilapin/runit, is not well-suited for what’s coming.

1. Pretty original ugh!

2. I can even say famous for some of them.

Assertion reporting

2022-02-16T00:00:00+01:00

This post is part of an ongoing series about tests in Rust:

Little recap

In the last post, we basically designed Assertion so we would use it like this.

let some_value = String::from("foo") + "bar";
Assertion::from(&some_value).is_equal_to("foobar");

or, since we provided a short macro for building Assertion.

let some_value = String::from("foo") + "bar";
assert_that!(some_value).is_equal_to("foobar");

And remember, we can chain assertions too. For example, let’s say we know our function life() should never return 0 nor 42.

let life = life();
assert_that!(life)
    .is_not_equal_to(0)
    .is_not_equal_to(42);

So far, the actual assertion will panic if one of the condition is not met. But since we’re chaining assertions, it might actually be useful to know the results for the entire chain instead of failing on the first incorrect one.

Storing the results

This should actually be pretty easy to store all the results since we’re having the Assertion object all along. We can stick a Vec into Assertion. This Vec will contain a Report that needs a few information:

did the assertion pass?
what was the test (so we can report to the end-user)?
what was the tested value?
what was the compared value?

#[derive(Debug)]
struct Report {
    pass: bool,
    condition: String,
    value: String,
    compared: String,
}

#[derive(Debug)]
pub struct Assertion<'v, V>
where
    V: std::fmt::Debug,
{
    value: &'v V,
    reports: Vec<Report>,
}

We can now implement is_equal_to() as follows.

pub fn is_equal_to<C>(mut self, compared: C) -> Self
where
    V: PartialEq<C> + std::fmt::Debug,
    C: std::fmt::Debug,
{
    let value: &'v V = self.value;
    let pass = value.eq(&compared);
    let reporter = crate::Report {
        pass,
        condition: "be equal to".to_string(),
        value: format!("{:?}", value),
        compared: format!("{:?}", compared),
    };
    self.reports.push(reporter);
    self
}

There is a few things going on. First of all, we changed the signature of the function is_equal_to so it now takes ownership of self. This will allow us to mutate it.^[1]

We also transformed value and compared into their Debug formatted string.

And know, we can try to run the following to see how it goes.

let life = life();
assert_that!(life).is_not_equal_to(42);

And it passes. Great! Hmm, just to be sure, does this fails?

let life = 42;
assert_that!(life).is_not_equal_to(42);

What, it pass too? Something is wrong. And indeed, remember how we had the following in is_equal_to() implementation before… but we do not have it anymore!

assert!(t.eq(&expected));

This is the line that made the test panic or not, and therefore the test pass or fail.

Failing again

^[2] Where do we insert the assert!() back into our code? We cannot put it in is_equal_to nor is_not_equal_to as this would defeat the purpose of not failing early. However, there is one piece of code that is always executed when a object goes out of scope: Drop.

Let’s try it.

impl<V> std::ops::Drop for Assertion<'_, V>
where
    V: std::fmt::Debug,
{
    fn drop(&mut self) {
        let nb_fails = self.reports.iter().filter(|r| r.pass == false).count();
        let message = self.reports.iter().fold(String::from("\n"), |msg, r| {
            let emoji = if r.pass { '✓' } else { '⨯' };
            msg + &format!(
                "\t{} should be {} `{}`, was `{}`\n",
                emoji, r.condition, r.compared, r.value,
            )
        });
        assert!(nb_fails == 0, "{}", message);
    }
}

Does it work? Let try back the following code.

let life = 42;
assert_that!(life).is_not_equal_to(42);

Here the output result. It does work!

stderr:
thread 'main' panicked at '
    ⨯ should not be equal to `42`, was `42`
'

More information in the report

We can add a little more useful information about where the assertion failed thanks to the file!() and line!() macros.

Let’s make these modifications to Assertion.

#[derive(Debug)]
struct Location {
    file: String,
    line: u32,
}

#[derive(Debug)]
pub struct Assertion<'v, V>
where
    V: std::fmt::Debug,
{
    value: &'v V,
    location: Option<Location>,
    reports: Vec<Report>,
}

impl<'v, V> From<&'v V> for Assertion<'v, V>
where
    V: 'v + std::fmt::Debug,
{
    fn from(value: &'v V) -> Self {
        Assertion {
            value,
            location: None,
            reports: Vec::new(),
        }
    }
}

impl<'v, V> Assertion<'v, V>
where
    V: 'v + std::fmt::Debug,
{
    pub fn with_location(mut self, file: &str, line: u32) -> Self {
        self.location = Some(Location {
            file: file.to_owned(),
            line,
        });
        self
    }
}

Note that file!() and especially line!() must be invoked at the exact place where the Assertion is built. If we stick them in the implementation of From above, this would not work because it would be expanded to our library file and line, not the one of the end-users. Once again, we need to rely on macros which will be expanded on-site and give use the correct result. Let’s modify assert_that.

#[macro_export]
macro_rules! assert_that {
    ($value:expr) => {{
        runit_assertions::Assertion::from(&($value)).with_location(file!(), line!())
    }};
}

Now, we can modify the Drop implementation like this.

impl<V> std::ops::Drop for Assertion<'_, V>
where
    V: std::fmt::Debug,
{
    fn drop(&mut self) {
        let fails = self.reports.iter().filter(|r| r.pass == false).count();
        let message = if let Some(location) = &self.location {
            format!("\n{}:{}\n", location.file, location.line)
        } else {
            String::from("\n")
        };
        let message = self.reports.iter().fold(message, |msg, r| {
            let emoji = if r.pass { '✓' } else { '⨯' };
            msg + &format!(
                "\t{} should be {} `{}`, was `{}`\n",
                emoji, r.condition, r.compared, r.value,
            )
        });
        assert!(fails == 0, "{}", message);
    }
}

Let run some tests again.

let life = 42;
assert_that!(life)
    .is_not_equal_to(0)
    .is_not_equal_to(42);

The output now looks like this.

stderr:
thread 'main' panicked at '
runit-assertions/src/lib.rs:66
    ✓ should be not equal to `0`, was `42`
    ⨯ should be not equal to `42`, was `42`
'

Some notes

I found out this little exercises pretty cool. Especially the need trick about displaying the file and line. There is however and drawback with this feature: it will not work for RustDoc, or at least, the line number will be wrong. Test it, you’ll see!

Again, you can browse the current implementation in woshilapin/runit.

1. We could have used &mut self instead of self, but there is no good reason to not consume it.

2. I like this title!

Equality assertions

2022-02-14T00:00:00+01:00

This post is part of an ongoing series about tests in Rust:

Equality for other types

In the possible improvements mentioned in the last post, I mentioned splitting NumberAssertions into 2 functionalities:

comparing equality of 2 objects (is_equal_to, is_not_equal_to)
comparing order of 2 objects (is_greater_than, is_less_than)

The funny thing is: we don’t need a new trait for these, they already exist and are called PartialEq et PartialOrd.

Let’s focus on PartialEq for the rest of this post.

What about chaining?

In the previous post, the API was allowing to chain multiple assertions. For example, this should be possible.

some_vec.has_length(3).contains("foo").contains("bar");

However, we do not have the NumberAssertions trait anymore that can be pass from assertion to the next one. So let’s create a struct that will refer to the tested value and might also contains more useful information that may be printed out for test results. For now, let’s do it simple and just retain information about the tested value.

struct Assertion<'v, V> {
    value: &'v V,
}

Construct an `Assertion`

Let’s create a quick constructor for this new struct and name it assert_that. This should allow us to write the following.

assert_that(1 + 1).is_equal_to(2);

`assert_that` as a function

How about a simple function to start.

pub fn assert_that<V>(value: V) -> Assertion<'_, V> {
    Assertion { value }
}

Oh wait, this consumes the value, we probably don’t want that. And it doesn’t work anyway.

error[E0106]: missing lifetime specifier
  --> runit-assertions/src/lib.rs:46:46
   |
46 | pub fn assert_that<V>(value: V) -> Assertion<'_, V> {
   |                                              ^^ expected named lifetime parameter
   |
= help: this function's return type contains a borrowed value with an elided lifetime, but the lifetime cannot be derived from the arguments
help: consider using the `'static` lifetime
   |
46 | pub fn assert_that<V>(value: V) -> Assertion<'static, V> {
   |                                              ~~~~~~~

For more information about this error, try `rustc --explain E0106`.

Let’s fix it…

pub fn assert_that<V>(value: &V) -> Assertion<'_, V> {
    Assertion { value }
}

Argh, it doesn’t work either.

error[E0308]: mismatched types
 --> src/lib.rs:44:13
  |
5 | assert_that(1 + 1).is_equal_to(2);
  |             ^^^^^
  |             |
  |             expected reference, found integer
  |             help: consider borrowing here: `&(1 + 1)`
  |
  = note: expected reference `&_`
                  found type `{integer}`

Ok, we do not want to take ownership but typing &(1+1) intead of 1+1 would make our API not easy to use. Can we do better? I believe we can thanks to macros.

`assert_that` as a macro

Let’s try it.

macro_rules! assert_that {
    ($value:expr) => {{
        runit_assertions::Assertion { value: &($value) }
    }};
}

Oh, but that wouldn’t work because value is a private field. We could make the field pub but no reason to expose it only for the constructor. There is actually another simple fix.

impl<'v, V> From<&'v V> for Assertion<'v, V>
where
    V: 'v,
{
    fn from(value: &'v V) -> Self {
        Assertion { value }
    }
}

macro_rules! assert_that {
    ($value:expr) => {{
        runit_assertions::Assertion::from(&($value))
    }};
}

Here we are, this finally works! Back to business after this little constructor’s digression.

Implementing `is_equal_to`

Let’s implement is_equal_to for Assertion now.

impl<'v, V> Assertion<'v, V> {
    pub fn is_equal_to(&self, expected: V) -> &Self
    where
        V: PartialEq,
    {
        let t: &V = self.value;
        assert!(t.eq(&expected));
        self
    }
}

The problem of that implementation? It doesn’t allow to do the following.

let s: String = some_function_under_test();
assert_that!(s).is_equal_to("foobar");

Indeed, "foobar" is of type &str but s is of type String. But I do remember an API of std that does exactly that: HashMap::get(). For example, the following does compile, even if the keys of the HashMap are String.

let mut map = HashMap::<String, usize>::default();
map.insert(String::from("foo"), 42);
let _ = map.get("foo");

This works thanks to the trait Borrow: with it, you can have a String and borrow it as if it was a &str.

Will Borrow help in our case? Well, yes and no. Ideally, I believe we should be able to support all of the following combinations.

assert_that!("foo").is_equal_to("foo");
assert_that!(String::from("foo")).is_equal_to("foo");
assert_that!("foo").is_equal_to(String::from("foo"));
assert_that!(String::from("foo")).is_equal_to(String::from("foo"));

And I did try multiple different combinations using Borrow, Into, or AsRef… but none of them worked. Until I realized something. Will you find the important part in the following fixed implementation?

pub fn is_equal_to<E>(&self, expected: E) -> &Self
where
    V: PartialEq<E>,
{
    let t: &V = self.value;
    assert!(t.eq(&expected));
    self
}

Of course you did spot that a new generic E was introduced. This is mandatory if we want to support a different type from V. However, the important piece is in the bounds of V: it went from V: PartialEq to V: PartialEq<E>. Yes, this single change will do the trick because String does implement PartialEq<String>, PartialEq<&str>, PartialEq<str> and even PartialEq<Cow<'a, str>>.

Conclusion

In the end, we were able to implement something that allows us to write the following assertions and I find it pretty nice so far.

assert_that!(1 + 1).is_equal_to(2);
let foobar = String::from("foobar");
assert_that!(foobar).is_equal_to("foobar");   // String == &str
assert_that!("foobar").is_equal_to("foobar"); // &str == &str
assert_that!("foobar").is_equal_to(&foobar);  // &str == &String
assert_that!("foobar").is_equal_to(foobar);   // &str == String

Looks good, isn’t it? You can browse the result of this implementation in woshilapin/runit.

`cargo-sonar`

2021-11-17T00:00:00+01:00

Announcing `cargo-sonar`

For a few days, I’ve been working on a small project called cargo-sonar and now is the time to make it public.

Why?

Sonar ^[1] is a great tool for managing the quality of your software. Although, you might disagree if you’ve been using it 10 years ago!

However, our beloved language Rust has not been supported by Sonar (although there have been discussion about it).

In a company, Sonar is a great tool for centralizing all metrics about code quality. So not having Rust’s support is another reason a company could rule out Rust as a technical solution (a small one but still a valid one).

The community already provided a great plugin for SonarQube called sonar-rust. Even if the maintenance of the code has not been very active in the last few months, the plugin just works fine. But there is one trick, it has to be installed server-side. If you want to manage your own in-house instance of Sonar, that’s the go-to solution. But sonarcloud.io who provides hosted instances of Sonar doesn’t allow custom plugins to be installed. So how do we do it?

How?

Well, Sonar actually have an API to provide generic issues. How about we use the output from clippy, transform it into this generic issue format, then send it to Sonar? That’s exactly what cargo-sonar is made for… and a bit more.

Setting up `cargo-sonar`

The best example of a working setup of cargo-sonar is probably the cargo-sonar project itself. You can see the result on sonarcloud.io directly. But let me show you the big lines as a starter.

Generate `clippy` output

First of all, you’ll have to generate clippy output in JSON format. You can even activate clippy::pedantic in this case to have maximum feedback.

cargo clippy --all-features --all-targets --message-format=json -- --warn clippy::pedantic > clippy.json

Convert into Sonar generic issues

Here is where cargo-sonar do its work.

cargo install cargo-sonar
cargo sonar --clippy

This is going to produce a file sonar.json from the clippy.json file ^[2].

Push the generic issues to sonarcloud.io

You’ll need an account on sonarcloud.io and a existing project. To push the generic issues, we’re going to use sonar-scanner.

Once you’ve installed sonar-scanner, you’ll have to create a sonar-project.properties file. I’ll take cargo-sonar as an example.

sonar.organization=woshilapin
sonar.projectKey=woshilapin_cargo-sonar
sonar.projectName=cargo-sonar
sonar.sources=.
sonar.host.url=https://sonarcloud.io
sonar.sourceEncoding=UTF-8
sonar.externalIssuesReportPaths=sonar.json
sonar.login=the_token_of_the_project

Do not forget to replace the sonar.login property with your project’s token. Did you see the sonar.externalIssuesReportPaths property refering to our sonar.json file. Now, you can just run sonar-scanner and go see on sonarcloud.io the result.

What’s more?

Well, clippy is nice but we have tons of tools in the Rust ecosystem. How about seing CVE in Sonar? How about outdated dependencies? Or licenses incompatibilities?

With the same logic, I’ve been implementing conversion into generic issues for the following tools.

`cargo-audit`

cargo audit --json > audit.json
cargo sonar --audit

`cargo-deny`

cargo deny --format json check 2> deny.json
cargo sonar --deny

`cargo-outdated`

cargo outdated --depth 1 --format json --workspace > outdated.json
cargo sonar --outdated

What’s next

The project is pretty young. Not a lot of tests, the documentation can definitely be better, the code is made of huge functions and could probably have a better architecture… but it works. I’m hoping for some feedbacks to better focus on the next steps. You can post on issues, or on forums like URLO or Sonar, I’m woshilapin on both of them.

1. I’m going to refer to Sonar for both SonarQube and Sonarcloud

2. You can change the filename of the clippy output with --clippy-path.

Dynamic iterators

2020-09-21T00:00:00+02:00

Introduction

If you ever tried to return an Iterator from a function and had difficulties with it, this article might help.

Let’s introduce a context first. Imagine for a minute you’re trying to define a trait to list the colors of anything. For example, you could implement this for vegetables, logos or even flags.

You’ll probably define the trait like this.

#[derive(Clone, Copy, Eq, Hash)]
struct Color(u8, u8, u8);
trait Colors {
	fn colors(&self) -> Vec<Color>;
}

Very simple right? But what about the zero-cost abstraction that Rust embraces. Why would we want to return a Vec<Color> if when we use it, we only need the first one for example? No need to allocate memory space for an entire Vec.

First try: return an `Iterator` type

Fine, let’s use an Iterator instead: the one returned by Vec::iter().

trait Colors<'a> {
	fn colors(&'a self) -> std::slice::Iter<'a, Self>;
}

At this point, we are enforcing which type of Iterator any implementor of Colors would need to use. If you don’t see the problem, let’s understand what is wrong with it before going forward.

Imagine the following implementation for struct Flag.

struct Flag {
	country: String,
	colors: Vec<Color>,
}
impl Colors<'a> for Flag {
	fn colors(&'a self) -> std::slice::Iter<'a, Color> {
		self.colors.iter()
    }
}

Everything looks fine so far. But why do we use a Vec<Color> for the list of colors. We shouldn’t have twice the same color in this list so let’s replace Vec<Color> with HashSet<Color> (Eq and Hash are implemented on Color).

You will get the following compilation error because self.colors.iter() doesn’t produce a std::slice::Iter but a std::collections::hash_set::Iter.

error[E0308]: mismatched types
  --> src/lib.rs:15:9
   |
15 |         self.colors.iter()
   |         ^^^^^^^^^^^^^^^^^^ expected struct `std::slice::Iter`, found struct `std::collections::hash_set::Iter`
   |
   = note: expected struct `std::slice::Iter<'a, Color>`
              found struct `std::collections::hash_set::Iter<'_, Color>`

Second try: associated type

Let’s change a bit our trait definition so the implementor can decide the Iterator type.

trait Colors<'a> {
    type ColorsIter: Iterator<Item = &'a Color>;
    fn colors(&'a self) -> Self::ColorsIter;
}

And then the implementation would become the following.

impl<'a> Colors<'a> for Flag {
    type ColorsIter = std::collections::hash_set::Iter<'a, Color>;
    fn colors(&'a self) -> Self::ColorsIter {
        self.colors.iter()
    }
}

Perfect, we’re done. Are we, really? Something itches me. Indeed, we basically push the complexity to the implementor. If I could, I’d like to have an API as simple as possible. On top of that, the complexity here might actually become close to a nightmare. Let’s say for example that Colors trait is documented as follow “list all the colors of an object (black is not considered a color)”.

But you’re the implementor of struct Flag and you do not really care of the consideration of some scientists to consider that black is not a color: some flags contains black and you have to store it. But that’s fine, you can still implement it by filtering out the black.

impl<'a> Colors<'a> for Flag {
    type ColorsIter = std::collections::hash_set::Iter<'a, Color>;
    fn colors(&'a self) -> Self::ColorsIter {
        self.colors
            .iter()
            .filter(|color| color.0 != 0 && color.1 != 0 && color.2 != 0)
    }
}

Here is the compilation error you’ll get.

error[E0308]: mismatched types
  --> src/lib.rs:17:9
   |
16 |       fn colors(&'a self) -> Self::ColorsIter {
   |                              ---------------- expected `std::collections::hash_set::Iter<'_, Color>` because of return type
17 | /         self.colors
18 | |             .iter()
19 | |             .filter(|color| color.0 != 0 && color.1 != 0 && color.2 != 0)
   | |_________________________________________________________________________^ expected struct `std::collections::hash_set::Iter`, found struct `std::iter::Filter`
   |
   = note: expected struct `std::collections::hash_set::Iter<'_, _>`
              found struct `std::iter::Filter<std::collections::hash_set::Iter<'_, _>, [closure@src/lib.rs:19:21: 19:73]>`

Of course, using .filter() doesn’t get you a std::collections::hash_set::Iter anymore but a std::iter::Filter. Lucky for you, the compiler is nice and tell you the type you need to use… or is it? Did you notice the part that looks like [closure@src/lib.rs:19:21: 19:73]? How do you define that?

Either you’re experienced enough and know how to get out of this situation, or you’re not and feel like maybe, you’re not going to use this Colors trait afterall (read fourth try below, it might help!). In any case, I hope you’re convinced our API leaked some bad complexity to the implementor. Can we do better?

Third try: `impl Iterator`

Since Rust 1.26, we can use the impl keyword in function parameters and return types. Let’s try it.

trait Colors<'a> {
    fn colors(&'a self) -> impl Iterator<Item = &'a Color>;
}

impl<'a> Colors<'a> for Flag {
    fn colors(&'a self) -> impl Iterator<Item = &'a Color> {
        self.colors
            .iter()
            .filter(|color| color.0 != 0 && color.1 != 0 && color.2 != 0)
    }
}

Well, here is what the compiler will tell you.

error[E0562]: `impl Trait` not allowed outside of function and inherent method return types
 --> src/lib.rs:6:28
  |
6 |     fn colors(&'a self) -> impl Iterator<Item = &'a Color>;
  |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

And indeed, if we jump to the RFC1522 that introduced this nice behavior, there is a limitation (might be lifted in the future but not now).

`impl Trait` may only be written within the return type of a freestanding or
inherent-impl function, not in trait definitions or any non-return type
position. They may also not appear in the return type of closure traits or
function pointers, unless these are themselves part of a legal return type.

Fork! It seems like the ideal solution but is not yet supported in the language.

Fourth try: `Box<dyn Iterator>`

Associated types were not a nice solution because we let the implementor deal with the complexity. But maybe there is a solution not so complex for the implementor. We know we want to return something that implements Iterator. Trait-objects might be the solution here.

Let’s get back to our previous definition using an associated type with a little tweak.

trait Colors<'a> {
    type ColorsIter: Iterator<Item = &'a Color> + 'a;
    fn colors(&'a self) -> Self::ColorsIter;
}

Note that since we iterate over references of colors, the Iterator itself must live at least as long as these references, which explains why we need to add + 'a.

Then we now can use some boxed trait-object to solve our problems.

impl<'a> Colors<'a> for Flag {
    type ColorsIter = Box<dyn Iterator<Item = &'a Color> + 'a>;
    fn colors(&'a self) -> Self::ColorsIter {
        Box::new(
            self.colors
                .iter()
                .filter(|color| color.0 != 0 && color.1 != 0 && color.2 != 0),
        )
    }
}

This works just fine. Using trait-object, the implementor don’t need to know the exact Iterator type, just that it implements Iterator.

Fifth try: wrapping `Box<dyn Iterator>`

Are we not done here? Well, here is me splitting hairs. Sure we’ve found something manageable but can we simplify a bit the life of a potential implementor by hiding some of these Box and dyn? How about we create a type to hide this complexity.

pub struct DynIter<'iter, V> {
	iter: Box<dyn Iterator<Item = V> + 'iter>,
}

And then we need to implement Iterator on it.

impl<'iter, V> Iterator for DynIter<'iter, V> {
    type Item = V;
    fn next(&mut self) -> Option<Self::Item> {
        self.iter.next()
    }
}

Let’s also create a new() method to remove some of the boilerplate of creating a DynIter.

impl<'iter, V> DynIter<'iter, V> {
    pub fn new<I>(iter: I) -> Self
    where
        I: Iterator<Item = V> + 'iter,
    {
        Self {
            iter: Box::new(iter),
        }
    }
}

And here we are, no more dyn or Box.

impl<'a> Colors<'a> for Flag {
    type ColorsIter = DynIter<'a, &'a Color>;
    fn colors(&'a self) -> Self::ColorsIter {
        DynIter::new(
            self.colors
                .iter()
                .filter(|color| color.0 != 0 && color.1 != 0 && color.2 != 0),
        )
    }
}

Conclusion

Since I’ve found out that the above solution could be reusable, I actually made a crate out of it: dyn-iter. Please report issue or improvement in the project repository.

I hope you learned something. At least, I’ve learned a lot by trying to make all of this work. I’m sure there is still some hidden corners that I don’t master yet. If you found any mistake in what I’m saying, or some ideas to improve this blog post, please let me know on the blog repository.

Take second try and third try, shake, and here we go

After the first publication of this article, chrysn provided some more information about RFC2515 that actually brings the impl keyword into associated types. At the time of writing ^[1], this is already available in nightly compiler.

#![feature(type_alias_impl_trait)]

#[derive(Clone, Copy, PartialEq, Eq, Hash)]
struct Color(u8, u8, u8);

struct Flag {
    country: String,
    colors: Vec<Color>,
}

trait Colors<'a> {
    type ColorsIter: Iterator<Item = &'a Color>;
    fn colors(&'a self) -> Self::ColorsIter;
}

impl<'a> Colors<'a> for Flag {
    type ColorsIter = impl Iterator<Item = &'a Color>;

    fn colors(&'a self) -> Self::ColorsIter {
        self.colors
            .iter()
            .filter(|color| color.0 != 0 && color.1 != 0 && color.2 != 0)
    }
}

Once this is stable, I’ll pretty much forget about dyn-iter crate and use this new solution instead.

1. Written on October 24th, 2020

Fluent assertions for numbers in Rust

2020-05-21T00:00:00+02:00

This post is part of an ongoing series about tests in Rust:

Introduction

Following up on my previous blog post about the testing ecosystem in Rust, I decided to try something on my own. I need to make a disclaimer right away: even if I already evaluated spectral as a nice candidate to start from, I decided to start from scratch because I want to discover all the tiny details by myself.

In this blog post, I’ll talk about my very first attempt to write some assertions for numbers.

API driven design

Let’s start by talking about the API we want. How about we start with this trait.

trait NumberAssertions {
	fn is_equal_to(&self, number: &Self);
	fn is_greater_than(&self, number: &Self);
	fn is_less_than(&self, number: &Self);
}

I know there is a bunch more we could write like is_not_equal_to, is_greater_or_equal_than, etc. but they can all be derived from the three above functions.

Note	The functions don’t return a value and you might be wondering why? We’re writing a framework for tests. If the `self` “is not greater than” some other value, we can `panic!()`, no need to return a `bool`.

With the above trait, and if we implement it on standard numbers (i32, u8, etc.), we should be able to write tests like this.

#[test]
fn it_works() {
	2u8.is_greater_than(1);
}

In the future code samples, I will simply ignore the redundant #[test] and function wrapping.

Chaining assertions

I see one first problem with the previous example, this API never mentions the concept of an assertion. For now, let’s do something really dumb.^[1]

fn assert_that<T>(provided: T) -> T {
	provided
}

assert_that(2u8).is_greater_than(1);

Great. But how would I test that a number is between 2 values?

assert_that(2u8).is_greater_than(1);
assert_that(2u8).is_less_than(3);

How about we provide the ability to chain calls.

assert_that(2u8).is_greater_than(1).is_less_than(3);

To do this, we can change the trait to something like.

trait NumberAssertions {
	fn is_equal_to(&self, number: &Self) -> &Self;
	fn is_greater_than(&self, number: &Self) -> &Self;
	fn is_less_than(&self, number: &Self) -> &Self;
}

Owned versus referenced value

There is another problem. Let’s imagine we do 2 different computations that should result in the same expected value.

let expected = 78648u64;
assert_that(compute_1()).is_equal_to(expected);
assert_that(compute_2()).is_equal_to(expected);

In this case, we’re lucky because u64 is Copy, otherwise, we would have a compile error.^[2] But if expected was not Copy, we probably want to be able to take the parameter as a reference or as a value.

let expected = 78648;
assert_that(compute_1()).is_equal_to(&expected);
assert_that(compute_2()).is_equal_to(expected);

The std::borrow::Borrow trait can help here.

trait NumberAssertions {
	fn is_equal_to<B>(&self, number: B) -> &Self where B: Borrow<Self>;
	fn is_greater_than<B>(&self, number: B) -> &Self where B: Borrow<Self>;
	fn is_less_than<B>(&self, number: B) -> &Self where B: Borrow<Self>;
}

Constraints

Finally, we forgot some constraints on Self to compare numbers. Let’s add PartialEq and PartialOrd as needed.

trait NumberAssertions {
	fn is_equal_to<B>(&self, number: B) -> &Self
	where
		Self: PartialEq,
		B: Borrow<Self>;
	fn is_greater_than<B>(&self, number: B) -> &Self
	where
		Self: PartialOrd,
		B: Borrow<Self>;
	fn is_less_than<B>(&self, number: B) -> &Self
	where
		Self: PartialOrd,
		B: Borrow<Self>;
}

Implementation

We can now implement a simple version for every number types.

impl<T> NumberAssertions for T {
	fn is_equal_to<B>(&self, number: B) -> &Self
	where
		B: Borrow<Self>,
		Self: PartialEq,
	{
		assert!(self.eq(number.borrow()));
		self
	}
	fn is_greater_than<B>(&self, number: B) -> &Self
	where
		B: Borrow<Self>,
		Self: PartialOrd,
	{
		assert!(self.partial_cmp(number.borrow()) == Some(std::cmp::Ordering::Greater));
		self
	}
	fn is_less_than<B>(&self, number: B) -> &Self
	where
		B: Borrow<Self>,
		Self: PartialOrd,
	{
		assert!(self.partial_cmp(number.borrow()) == Some(std::cmp::Ordering::Less));
		self
	}
}

We can also add some provided methods in the trait like is_between(n1, n2).

trait NumberAssertions {
	// other functions not displayed here
	fn is_between<B1, B2>(&self, min: B1, max: B2) -> &Self
	where
		Self: PartialEq,
		B1: Borrow<Self>,
		B2: Borrow<Self>,
	{
		self.is_greater_than(min).is_less_than(max)
	}
}

Testing the implementation

Let’s write a bunch of tests to see if everything is working as expected.

#[test]
fn greater_than() {
	assert_that(2u8).is_equal_to(2);
	assert_that(2i32).is_greater_than(1);
	assert_that(2.0f64).is_greater_than(1.0);
	assert_that(1u64).is_less_than(2);
	assert_that(1.0f32).is_less_than(2.0);
	assert_that(2i32).is_between(1, 3);
	assert_that(2u16).is_greater_than(0).is_less_than(100);
}

#[test]
#[should_panic]
fn greater_than_panics() {
	assert_that(2i32).is_greater_than(4);
}

Conclusion

That’s a great start… but I already see room for improvement.

“Is not equal” and friends

First of all, I did not provide an implementation for is_not_equal_to()? Did you think I forgot? Well, in fact, I cannot implement is_not_equal_to() because is_equal_to() panics in case of failure. So basically, we would need to not panic when is_equal_to() panics which is not possible.^[3]

Do I need to go back to functions that return bool? But if I do, what about chaining? Another solution would be to let the implementors implement the functions but that means more work for each implementor. Ideally, someone should be able to implement NumberAssertions easily for a new type. Not yet sure how to overcome this problem.

More functions

In this blog post, I focused on a very simple API about equality and comparison. I’m sure we could imagine more useful assertions functions like is_multiple_of(), is_zero(), is_not_a_number(), is_prime(), etc. I’ll probably add more functions later, once I’m satisfied the code architecture.

Equality for other types

is_equal_to() is part of the trait NumberAssertions. But equality is not exclusive to numbers, we could also compare String, and any kind of object that implements PartialEq. I might need to split into two parts the NumberAssertions trait.

Naming test

When using assert!(), we can name the test by doing assert!(1 == 1, "Test dumb equality"). Our current solution forbids that. When looking into spectral, we can see that assert_that(provided) doesn’t return provided but a wrapper type named Spec<T>. This Spec can contain more information about the current assertion (the name of the test or the location).

Feedbacks

Implementation can be found at https://gitlab.com/woshilapin/runit. I’d be happy to hear feedbacks, suggestions, or even better, contributors if you’re interested in this project.

1. We could also rename all our functions with assert_is_equal_to() but that would make the functions much longer, repetitive and less fluent.

2. First call to is_equal_to() takes ownership of expected, second call will fail the compilation

3. I might be able to use std::panic::catch_unwind but that would mean running assertions in another thread which is not an option for tests.

vim running Rust tests

2020-05-21T00:00:00+02:00

Introduction

If you ever used VS Code to edit Rust source code, you might already know you can launch just the test you care about.

But I love too much Vim to convert to VSCode. And yes, I know there is a Vim plugin but these emulation plugins are never complete and I always end up using vim shortcuts that doesn’t exist in the plugin. It annoys me.^[1]

However, there is a bunch of nice things in VSCode like running the test you want, without running the entire suite test.

Selecting tests with `cargo`

First of all, let see how to run a test and only the tests you want.

If you have a lib.rs with the following test.

#[cfg(test)]
mod tests {
	use super::*;
	mod addition_tests {
		use super::*;
		#[test]
		fn add_works() {
			assert_eq!(1 + 1, 2);
		}

		#[test]
		#[should_panic]
		fn add_panics() {
			assert_ne!(1 + 1, 2);
		}
	}
}

Then you can run the add_work() test function with

cargo test --lib -- --exact tests::addition_tests::add_works

Or run all the tests in this file with

cargo test --lib -- tests::addition_tests

And it works similarly for integration tests.

cargo test --test addition

will run all the tests in tests/addition.rs and

cargo test --test addition -- --exact add_works

will only run the add_works() function in tests/addition.rs.

Let’s do it in vim

The algorithm is relatively simple. We start from the position of the cursor and try to deduce what test the user want to launch. If the cursor is inside a test function, let’s launch only this function. If the cursor is in a file that contains test, let’s launch all the tests of this file, etc.

Here is a simplified algorithm. First, we iterate over the blocks { … } going from the narrower block containing the cursor, to the larger block containing the cursor. During this iteration, we identify the block of the test function (function annotated with #[test]) if there is one, and all the wrapping modules. Then we can iterate on the file path to deduce the rest of the modules.

From the file path, we can also deduce if this is an integration test or not. This will help to choose between --lib or --test options.

You can find the code in my vim script here.

Conclusion

Now, you have a nice function that you can map to a shortcut. I use F6.

noremap <F6> <Esc>:call <SID>rust_execute_test()<Enter>

I will add an additional comment because we forget too often about that, vim documentation is awesome. Most of my problems in writing this script was solved through documentation. The only drawback of the documentation is to find the right section. ^[2]

1. I’ve had the same problem with IdeaVim in IntelliJ or with Vimium to name a few

2. As I am writing this blog post, I’m discovering the command :helpgrep which might have help me.

Hole is back

2020-05-17T00:00:00+02:00

Introduction

It has been about 6 years or so since I last wrote on this blog. Life happened I guess! But I once again have the envy to express myself: document things I experiment, publish experiences I do. Always about computer sciences, what else!

What changed?

Well, quite a lot changed in 6 years, professionally and personally but I’m not here to talk about my life. However, I can say that my topics of interest evolved and also narrowed in some ways. I’m still pretty much interested in a lot of things and here are the non-exhaustive list of things I’ve touch in the last years.

Javascript

With front-end and back-end, I experimented with a lot of different technologies build tools, automatic integration and deployment. I tried the async/await before it was wide-spread (with some exotic configuration of Babel transpiler). With it, I obviously tried some CSS frameworks too (LESS, PostCSS) as well as templating for HTML (like Jade). I tried to explore the HTML5 world as much as possible.

Docker

I also discovered Docker which I used quite a lot for development. I was not confident enough to use it for deployment in production most of the time but as a developer tool, it was wonderful to install different database version or some MQ tools without polluting my own OS. And I used Docker almost exclusively through Docker Compose.

Java

I love and don’t like this language at the same time. I love it so much because it has the richest ecosystem that I know. But I also don’t like it that much because, well, the language is not that fun to write. Maybe the only other language with such a rich ecosystem is C but in a much more chaotic way. In Java, you have tons of very complete tools for doing pretty much anything. Testing in Java is awesome. Build tools are quite complete (Maven might be slow and XML but you can pretty much do anything with it). I’d like to try Gradle at some point. And so much more, Junit5, asserting libraries like AssertJ, code quality and coverage with SonarQube (this tool is so excellent).

Rust

More recently, I’ve been able to find much more time to explore my curiosity about, I can say it now, this wonderful language and community that is Rust. And Rust ecosystem is actually the reason why at some point, I wanted to unearth this blog. I won’t talk much more about it since future blog post will probably talk about it.

Others

I also took a look at things like Go, Docker Swarm, Kubernetes, Gitlab CI, CodingGame, VueJS, WebAssembly, WebExtensions and I probably forgot more of them.

What else?

Well, I’ve changed the way I deploy this blog, using Gitlab CI. It’s now much easier than it used to be to write a new blog post because it’s only about adding a ASCIIDoc file into the repository, and Gitlab take care of the rest.

Conclusion

What, do we really need a conclusion? Well, see you in the next blog post!

Rust testing ecosystem

2020-05-17T00:00:00+02:00

This post is part of an ongoing series about tests in Rust:

Introduction

I’ve been working with Rust daily for a year now and generally, I have been very happy with this language and it’s ecosystem. Of course, the ecosystem is still pretty new, a lot of things are missing or incomplete but it’s only a five year’s language and I feel that for this age, its already pretty mature.

However, there is one domain in which I feel this language is still pretty poor in term of ecosystem: testing. And I love testing! No really, I know most people find the task of writing tests boring, annoying but this is a task I am generally enthusiast to do for multiple reasons:

I don’t trust my code, test is a fun way to automatically verify what I’ve done
I found out that in general, TDD helps me think better of a problem before implementing it
I believe that testable code is producing better code architecture

Testing in Rust language

Today in Rust, testing can be summarized in a few functionalities:

a #[test] macro
an assert!() macro
a test configuration option (used in #[cfg(test)])
a folder tests/ with cargo ^[1]
Documentation tests

Compared to other languages, it’s pretty similar: an assert command and the rest is to declare what and where is a test. The notable exception is Documentation Tests which is quite innovative and awesome.

In this post, I will focus on assertions. Indeed, assert!() is a necessary building block and a simple one, but it’s not really fun to use. Better API can be built on top of it.

In this post, I will not talk about mocking (there is a bunch of libraries around) or other types of testing like property testing (check quickcheck and proptest) or fuzzing.

Assertion

For asserting in tests, most languages provides libraries on top of the assert function in order to make it more fluent to write unit tests. I will cite two notable examples I’ve worked with in other languages (but there is much more libraries out there): AssertJ and ChaiJS.

Let’s take a look at some existing libraries in Rust ecosystem.

difference

This library is specialized in asserting equalities between &str. The quality is quite good and it also show a very nice diff output… but of course, it’s only about asserting equality of &str.

approx

This library is specialized in asserting floating numbers. Assertion on floating numbers is not as easy because of rounding errors. approx helps in asserting floating numbers that are almost equals!

totems

This library is the first with an intuitive yet simple syntax for asserting things like enum assertions (although only for Result and Option), tuples, and collections. It’s pretty simple but effective. However, a more fluent API would be nice.

There hasn’t been a release in a year and everything has been developed in a month by one developer. This is not promising in term of maintenance.

hamcrest-rust and hamcrest2

The former is not maintained anymore, the second being a fork. hamcrest2 is the effort of one developer and no release (no commit on the project either) has been done in the last 6 months. And also, but that might be a personal preference of me, writing assertions with Hamcrest is not pleasant: you have to wrap in parenthesis everything. See this example:

assert_that!(1, not(eq(2)));

expecttest

Another very nice and relatively complete library. However, it seems to me it has almost the same kind of problems than hamcrest2, you have to wrap most of things in parenthesis (although the syntax is slightly better), there is only one developer and last release has been 7 months ago. One thing to notice is that the developer explicitely flagged the project as passively maintained so maybe some contributions would revive the project?

spectral

This one provide a more fluent API. For example, you can write things like:

assert_that(&vec![1, 2, 3]).has_length(3);

On top of that, the output failure message is improved to show in a more structured way the expected and the produced value. The default printing is not always easy to read.

This library is actually really close to something I would look for but, there is few problems with it.

First of all, it’s far from complete, more assertions functions could be added. I actually tried to add some more in this PR but this leads me to my second point.

The library is not maintained anymore. Last release was three years ago and since this crate is owned by the only developer who implemented it, only a fork seems to be a way forward at this point.

Finally, I don’t see anything about chaining assertions in this library. For example, in AssertJ, you can do things like this.

assertThat("QXNzZXJ0Sg")
	.decodedAsBase64()
	.containsExactly("AssertJ".getBytes());
assertThat(throwable)
	.hasMessage("top level")
	.getCause()
	.hasMessage("cause message");

Conclusion

Ideally, something like spectral is close from what I would look for. Forking might be a solution. I actually started to improve some annoyments I had in a branch. Much more functionalities could be added to it. Here is a non-exhaustive list of things:

more APIs (take example on other libraries in other languages)
ability to chain calls
implement as features for other libraries (for example, a specific map implementation like vec_map could implement HashMapAssertions).
colorize output error messages

Of course, I might have missed other libraries. I mostly used crates.io to search with only a bunch of keywords.

Sources

crates.io, where you can find most of the libraries from this post (the only exception seems to be hamcrest-rust
Awesome Rust, which lists interesting project in Rust, especially the “Testing” section

1. There is also benches/ but most of what you need to use it is only available in nightly Rust, see test::bench

Displaying dates and time in Gnuplot

2014-08-01T00:00:00+02:00

Introduction

When you need to display some specific data in Gnuplot, it may be hard. However, for dates, it is easy, there is already a functionality.

For example, if you want to display date and hour base on a timestamp you have in your data, you can and that is what I will explain.

What we want?

The log file we want to display contains a abscissa which is a time information stored as a timestamp (number of seconds since the epoch). However, we do not want that this big number appears on the abscissa axis. We want the time to be displayed as “31/12 12h” (day and month then only the hour). Moreover, we want a axis which have tic every half-day (midnight and midday).

Specify the type of data

First of all, we will specify that the x axis will display time information.

set xdata time;

We will now define the format of the input data with the timefmt option. Gnuplot use a standard way to describe date and time (try help timefmt to have more information). For us, it is a timestamp so we will use the "%s" time format.

set timefmt "%s";

We also want to format the output time format (the one that will be displayed on the graphic) with the following: - %d for the day - %m for the month - %H for the hour

set format x "%d/%m %Hh";

Finally, we need to define the interval between to ticks in order to displayed ticks only on midnight and midday. For this, we need to make a small calculus.

The timestamp is expressed in seconds. The interval we are taking to is 12 hours. So we need to express 12 hours in seconds which is 43200.

set xtics 43200;

Conclusion

Do not forget to use help to find what possibilities you have for time format.

Gnuplot graphic with 2 different scales

2014-08-01T00:00:00+02:00

Introduction

Gnuplot is a nice tool to create graphics. However, sometimes, we need more specific functionalities we may think Gnuplot does not have. A deeper look into Gnuplot functionalities show us that it is a very powerfull tool!

Yesterday, I needed to create a graphic with 2 curves on it. The problem is that the first is below 1000 and the other may be over 5000. I need a two scales graphic with one scale on the left and the other scale on the right. How do we do that on Gnuplot?

What we want?

There is 2 datasets that we want to display on the same graphic. The scale of the two datasets would be different. Consequently, we will have a left axis with different label and tics than the right axis. Moreover, we will use different color in order to associate each axis to each curve: red for the left axis (with the first curve) and blue for the right axis (and the second curve). Finally, we will hide the legend since the colors should be enough to understand the datas.

Configure the first part of the graphic

The first part of the graphic is the basic part you would expect from a graphic: a left scale with a label and tics.

set yrange [0:1000];
set ylabel "Title on the left";
set ytics 100 nomirror textcolor lt 1;

First of all, we define the lower and upper bound of the first ordinate curve. The second point is to set the title of the axis. Finally, we specify the number of tics (each 100 units in this case). For this last step, we also specify the color of it because we want a red color for the left axis. The nomirror option specify that we want the tics only on the left axis (no mirrored tics on the right axis).

Configure the second part of the graphic

We will do nearly the same for the second graphic. The only thing to know is that for each y<something> option, there is also a y2<something> corresponding option for the right axis.

set y2range [0:8000];
set y2label "Title on the right";
set y2tics 1000 nomirror textcolor lt 3;

Note the change in the color of tics.

Hide the legend

Hiding the legend in Gnuplot is easy… as long as you know how the option is called!

unset key;

Drawing the plots

To draw the plots, we use the plot function. Most of the time, if you want to draw to curve on the same graphic, that’s because they have the same dataset in abscissa. That’s mainly imply that all the data are in the same file with more than 2 columns. For example, a file like this.

time	left	right
0.00	100.0	750.0
0.50	150.0	1000.0
1.00	225.0	1500.0
1.50	325.0	2225.0
2.00	450.0	3225.0
2.50	600.0	4500.0
3.00	775.0	6000.0

In Gnuplot, we can load a multicolumns data file and choose which column to display on which graphic. This is how we will plot our data which are stored in the file data.csv.

plot \
	"data.csv" using 1:2 linetype 1, \
	"data.csv" using 1:3 linetype 3 axes x1y2

We give the name of the file to the plot function. Then we specify the columns to use (always the first in this case since it is our abscissa). We also change the color of the curve with the linetype option. The last important thing is to specify which axis (and which scale) to use for which curve (x1y1 is the default).

Bonus: export as a file

To export your Gnuplot result as an image file, you need to add at the beginning of the script the following lines.

set terminal png size 1024,768 enhanced font "Helvetica,20"
set output 'output.png'

Conclusion

Here is the result.

You can find the Gnuplot script here.

Replace multi-lines pattern with sed

2014-06-27T00:00:00+02:00

Introduction

Let’s define this problem by an example. This is the kind of text I want to modify.

# include <stdio.h>
void main(int argc, char ** argv) {
	printf("Hello world!\n");
	printf("Version 2.0\n\
			You can use this program as you want.\n");
	return 0;
}

This is a C program but I want to use gettext to provide translated version of this program easily. Usually, the solution is to replace each string by a call to a function (the string being the key). I will not enter in details of gettext here but to resume, we want something like this as the result.

# include <stdio.h>
void main(int argc, char ** argv) {
	printf(_("Hello world!\n"));
	printf(_("Version 2.0\n\
				You can use this program as you want.\n"));
	return 0;
}

Note that each string has been nested into the function _() wich is the usual convention for gettext. If you want the quick solution, go directly to the end of this article.

Brief introduction to `sed`

To understand the explanations that will follow, you must at least understand the basics from sed. You can either read this section but I strongly encourage to read the official sed documentation which is well-written.

Spaces

The first thing is that sed will parse the lines one by one. Each line parsed is put inside the pattern space, then actions can be done on it, and finally the line is printed.

However, there is also a hold space where you can store informations. And if you read the sed documentation, you’ll see that there is actions which can copy from one space to the other. This will be fundamental for the multi-line pattern replacement since sed is mostly for line-by-line edition. Keep that in mind.

`sed` with a script file

There is two main way of running sed. You can either give arguments to the command-line or give a sed script. For simple commands, you may want to use command-line since it’s faster than creating a script file then giving it as an argument to sed. However, for complex commands, you may want to work in a script file, test it, improve it, correct it. For example, lets do a simple replacement/deletion and see how we can run it in one or the other way.

With command-line, you’ll do the replacement this way

	sed 's/PATTERN/TEXTIWANT/' -e '/OTHERPATTERN/d' my_file.txt

You can add multiple actions by appending -e for each new action.

For the script-file solution, you would create a file my_file.sed for example that will contains the actions.

	s/PATTERN/TEXTIWANT/
	/OTHERPATTERN/d

Note that the quotes are not needed in the script file. Now, you can run the script.

	sed -f my_file.sed my_file.txt

Actions on target

In sed, most of the possible actions can be restricted to specific lines. For example, in the previous section, we used the d action to delete lines. This action has been prefixed with the pattern that should trigger this action.

n will look for the n th line of the input text
$ refers to the last line of the input text
/PATTERN/ will look for lines containing PATTERN

You can also select ranges by using 2 of the previous possibilities separated with a comma. For example, the following will select and delete any line between a line containing BPATTERN and a line containing EPATTERN.

	/BPATTERN/,/EPATTERN/d

Group actions

Sometimes, you need to apply more than one action to the same selected line. In this case, you can nest the actions into { and }. For example, if between BPATTERN and EPATTERN, there is line you want to delete (line containing BAR) and others you want to modify (change PATT into OUT).

	/BPATTERN/,/EPATTERN/ {
		/BAR/d
		s/PATT/OUT/g
	}

Multi-lines pattern

Now, you should have the basics to understand what will follow. These are the steps to follow to make it possible:

Print all unconcerned line as it is
Put the line that contains the beginning of our pattern into the hold space
Append all lines that are not the beginning of our pattern into the hold space
Operate multiple operations only on our pattern and when the last line of the pattern is reached
- Copy what’s in hold space into the pattern space (should contain all the lines we need)
- Do the replacement
- Print the result

The pattern

The pattern is simple, we know where it begins and when it ends. The start of the pattern is /("/ and the end is /");/. Our pattern can be defined as a range of line like the following.

	/("/,/");/

Print lines not in the pattern

We know how to select our pattern. However, we don’t know how to select all lines that are not in the pattern. sed provides us a way to do that with the character !. Prefixing an action with this character will apply the actions on all lines that are not the selected pattern or range.

The action to print the current line is p. Printing the lines that are not in our pattern can be done like the following.

	/("/,/");/!p

Filling the hold space

To fill the hold space, there is 2 possible actions (there is a third one, x but we will not discuss it):

h will replace the hold space with the content of the pattern space (usually the current line).
H will append to the hold space what’s in the pattern space

When we reach the beginning of our pattern, we will call the h action. Else, we will append the current line to the hold space with H.

	/("/h
	/("/!H

In our example of 7 lnies, the hold space will successively have the following values (you may use g then l to see it).

# include <stdio.h>
# include <stdio.h>\nvoid main(int argc, char ** argv) {
\tprintf("Hello world!\\n");
\tprintf("Version 2.0\\n\
\tprintf("Version 2.0\\n\\\n\t\tYou can use this program as you want.\\n");
\tprintf("Version 2.0\\n\\\n\t\tYou can use this program as you want.\\n");\n\treturn 0;
\tprintf("Version 2.0\\n\\\n\t\tYou can use this program as you want.\\n");\n\treturn 0;\n}

Note that the backslash and the tabulation has been escaped with a backslash.

Do the replacement

Now, we want do the replacement when the last line of our pattern is reached. To confine this modification, we first nest all the following operations into the desired range.

	/("/,/");/ {
		...
	}

Then, when we reach the end of our pattern, we create a new block where the actions will take place.

	/("/,/");/ {
		/");/ {
			...
		}
	}

The actions will be the following. First, we restore (with g) the content of the hold space into the pattern space (where actions take place). Then we do the replacement. And finally, we print the result.

	/("/,/");/ {
		/");/ {
			g
			s/(\("[^"]*"\))/(_(\1))/g
			p
		}
	}

Resume

First, we have the following file main.c.

# include <stdio.h>
void main(int argc, char ** argv) {
	printf("Hello world!\n");
	printf("Version 2.0\n\
			You can use this program as you want.\n");
	return 0;
}

Then we have the main.sed script.

/("/,/");/!p
/("/h
/("/!H
/("/,/");/ {
	/");/ {
		g
		s/(\("[^"]*"\))/(_(\1))/g
		p
	}
}

We can run the following command.

	sed -f main.sed main.c

Then we should obtain the following result.

# include <stdio.h>
void main(int argc, char ** argv) {
	printf(_("Hello world!\n"));
	printf(_("Version 2.0\n\
				You can use this program as you want.\n"));
	return 0;
}

References

sed documentation
A (flashy) web page about multi-lines pattern replacement

Migrating OwnCloud from SQlite to PostgreSQL

2014-06-26T00:00:00+02:00

Introduction

SQLite is not a real database system. It’s more like a big file of text. Using SQLite to manage any kind of system can be dangerous when your database grows. The best advise is to loose a bit more time to learn how to set up a real database system like MySQL or PostgreSQL.

However, sometimes you started with a SQLite database. Then you may want to migrate to a real database system. Let’s do that.

In this article, a SQLite 3 database owncloud.db will be migrated into a PostgreSQL 9.1 database (with username owncloud and database name owncloud). The use case is migrating the database of a instance of OwnCloud 6.0.3 with basic functionalities. Some applications have also been installed and the only noticeable one is News (allows you to subscribe to RSS flux).

Export the SQLite database

This is a pretty easy step since sqlite3 tool can give you a dump of your database as SQL commands. Let’s put the output into a file dump.sql.

sqlite3 owncloud.db .dump > dump.sql

Seems easy, right? Let’s go to the next step.

Import the dump into PostgreSQL

The nomal way

Since SQL is standard, we can try to feed PostgreSQL with this dump.sql file with the following command.

psql -U owncloud -d owncloud < owncloud.db

If it works for you, great! However, as we will see, each database system… has its own SQL standard.

Converting the dump

There is a few things to adapt between SQLite and PostgreSQL. An entry point can be this article which explains what should be adapted. Let’s sum up everything that should be adapted:

Remove PRAGMA calls
Convert DATETIME into TIMESTAMP (DATETIME is for SQLite when TIMESTAMP seems to be the one for PostgreSQL, see documentation)
Remove UNSIGNED (every integer is unsigned in PostgreSQL, see documentation)
Remove AUTOINCREMENT then replace INTEGER with SERIAL (SERIAL is the autoincrement integer in PostgreSQL)
Convert LONGTEXT and CLOB into TEXT or VARCHAR (see documentation)
Convert BLOB into BYTEA (and you might need more work for these ones)
Convert boolean values from respectively 0 and 1 to '0' and '1' or TRUE and FALSE (see documentation)

Except the last 2 points that requires more attention, we can do everything with sed. Let’s do a sed script migrate.sed for automatically adapting our dump dump.sql.

# Remove every line starting with `PRAGMA'
/^PRAGMA/d
# Change types only in the scope of a table creation
/^CREATE TABLE/ {
	# Remove `UNSIGNED' keyword and spaces following
	s/\<UNSIGNED\> *//g
	# Replace `DATETIME' type (SQLite) with `TIMESTAMP' type (PostgreSQL)
	s/\<DATETIME\>/TIMESTAMP/g
	# `LONGTEXT' and `CLOB' replaced by `TEXT' (could use `VARCHAR' too)
	s/\<LONGTEXT\|CLOB\>/TEXT/g
	# Replace any integer with `AUTOINCREMENT' with `SERIAL'
	s/\<INTEGER\>\([^,]*\)[ ]*AUTOINCREMENT/SERIAL\1/g
	# Replace `BLOB' with `BYTEA' (binary blob in PostgreSQL)
	s/\<BLOB\>/BYTEA/g
}

Some notes about that script.

Note	`PRAGMA` are only specific commands specific for the database system. Obviously, if they have sense for SQLite, they will not for PostgreSQL. We may find equivalent `PRAGMA’ commands but I didn’t investigate.

Note	We want to replace types only for table creation. For example, we want to avoid to remove `UNSIGNED` everywhere it appears in the file `dump.sql` because it may appears in the contents of the database. `sed` allows us to restrict modification to only some specific lines.

Note	`AUTOINCREMENT` in SQLite is linked with an integer where in PostgreSQL, `SERIAL` replaces both.

Note	`BLOB` is a binary blob which may need to be decoded with the `decode()` function (something like `decode(value,"hex")`). This can be tricky but see next sections, you may find some help.

After these few notes about the sed script, let’s apply it.

sed -f migrate.sed dump.sql > dump-adapted.sql

`BLOB` and `BOOLEAN` adptations

This is a more tricky problem since we will need to replace or encapsulate value that will be inserted into our database (lines that starts with INSERT INTO). For this purpose, I made a awk script to parse lines with CREATE TABLE and transform them, for each table, into sed replacement command that will target only the needed field (for example, the third one if the third one is a boolean, and the fifth one if it’s a BLOB).

#!/bin/awk -f

# Before program
BEGIN {
}

# Program
{
	print "# " $0;
	header = 0;
	lastpatt = "";
	for(k=1; k<=NF; k++) {
		if($k ~ /BLOB|BOOLEAN/) {
			subin="\\(^INSERT INTO \042\\?" ENVIRON["DB"] "\042\\? VALUES(";
			subout = "";
			can_be_null=0;
			subnum=1;
			for(i=1; i<=NF; i++) {
				patt = "";
				if(($i !~ /KEY/) && (i != 1)) {
					subin = subin ",";
				}
				if($i ~ /INTEGER/) {
					patt = "[[:digit:]]\\+";
				}
				if($i ~ /NUMERIC/) {
					patt = "[[:digit:].-]\\+";
				}
				if(($i ~ /VARCHAR|(LONG)?TEXT|BLOB/)) {
					patt = "\047.*\047";
				}
				if(($i !~ /NOT NULL|KEY/)) {
					patt = "\\(NULL\\|" patt "\\)";
					subnum = subnum + 1;
				}
				if($i ~ /BLOB/) {
					if (i < k) {
						patt = "decode(" patt ",\047hex\047)";
					}
					if(i == k) {
						if(patt ~ /NULL/) {
							bdecnum = 1;
							decnum = subnum;
							adecnum = decnum + 1;
							patt = "\\)" patt "\\(";
						} else {
							bdecnum = 1;
							decnum = subnum + 1;
							adecnum = decnum + 1;
							patt = "\\)\\(" patt "\\)\\(";
						}
						subout = "\\" bdecnum "decode(\\" decnum ",\047hex\047)\\" adecnum;
					}
				}
				if($i ~ /BOOLEAN/) {
					patt = "[01]";
					if(i == k) {
						bdecnum = 1;
						decnum = subnum + 1;
						adecnum = decnum + 1;
						patt = "\\)\\(" patt "\\)\\(";
						subout = "\\" bdecnum "\047\\" decnum "\047\\" adecnum;
					}
				}
				if(($i !~ /KEY/)) {
					lastpatt = patt;
				}
				subin = subin patt;
				can_be_null = 0;
			}
			subin = subin ");$\\)";
			if(header == 0) {
				startpatt = ENVIRON["START_PATT"];
				endpatt = lastpatt ENVIRON["END_PATT"];
				print "/" startpatt "/,/" endpatt "/ {";
				print "/" startpatt "/h";
				print "/" startpatt "/!H";
				print "/" endpatt "/ {";
				print "g";
				header = 1;
			}
			print "s/" subin "/" subout "/g";
		}
	}
	print "p";
	print "}";
	print "d";
	print "}";
}

# After program
END {
}

It’s a pretty unreadable awk and sed collaboration so you may try to understand only if you needed to. Otherwise, you may want to try the bash script I wrote which will run the awk script to produce the sed script then run the sed script to adapt the dump file.

#!/bin/sh
# The original database
SRC_FILE=owncloud.db
# A list of SQL commands that should create a copy or the source database
DUMP_FILE=dump.sql
# A list of `sed` actions to convert the SQLite dump into PostgreSQL
SED_FILE=migrate.sed
# Awk program to create sed actions for the BYTEA
AWK_FILE=migrate.awk

if [ -f $DUMP_FILE ]
then
	rm $DUMP_FILE
fi
if [ -f $SED_FILE ]
then
	rm $SED_FILE
fi

sqlite3 $SRC_FILE .dump > $DUMP_FILE

function to_sed() {
  echo $1 >> $SED_FILE
}

to_sed '#!sed -f'
to_sed '/^PRAGMA/d'
to_sed '/^CREATE TABLE/ {'
to_sed 's/\<UNSIGNED\> *//g'
to_sed 's/\<DATETIME\>/TIMESTAMP/g'
to_sed 's/\<LONGTEXT\>/TEXT/g'
to_sed 's/\<INTEGER\>\([^,]*\)[ ]*AUTOINCREMENT/SERIAL\1/g'
to_sed 's/\<CLOB\>/TEXT/g'
to_sed 's/\<BLOB\>/BYTEA/g'
to_sed '}'

cat $DUMP_FILE | grep '^CREATE TABLE.*\<\(BLOB\|BOOLEAN\)\>'| while read LINE
do
	DB=$( echo $LINE | awk '{print $3}' | sed -e 's/"//g' -e "s/'//g" )
	START_PATT='^INSERT INTO "\?\<'$DB'\>"\?'
	END_PATT=');$'
	export DB START_PATT END_PATT
	echo $LINE | sed -e 's/^CREATE TABLE "\?\<'$DB'\>"\? (//g' -e 's/);$//g' -e 's/([^)]*)//g' | awk -F ' *, *' -f $AWK_FILE >> $SED_FILE
done

to_sed "s/decode(NULL,'hex')/NULL/g"
to_sed 'p'

sed -n -f $SED_FILE -i.bak $DUMP_FILE

Remember, you may need these scripts if you have either BOOLEAN and/or BLOB. If not, then you probably don’t wwant them and just use the simple sed script I provided in the previous section.

Conclusion

To conclude to this article, even if I do a lot of research and it tooks me time to write these scripts, I didn’t success well in migrating from SQLite to PostgreSQL. In the meanwhile, I learned a lot of interesting stuff about awk and sed scripting, about playing with PostgreSQL. I also learned that you want probably to use PostgreSQL or MySQL from the start and avoid any migration. This article may helps you to understand some important things about the migration process, to find information but it will probably not be enough to make it well. It’s can only be a starting point. Good luck!

References

Dumping SQLite database
Migrate from SQLite to PostgreSQL with a few useful links:

Installing Movim on PostgreSQL

2014-06-19T00:00:00+02:00

Introduction

Do you know Movim? You should take a look if you’re interesting in social network that are trying to take care of your privacy. With it, you own your data.

OK, that’s great… but that means that you must install it on your server. Let’s do it!

Thie article will explain how to install the last unstable version of movim, 0.8alpha12 at this time (but it should work with stable versions too), on a Debian server runned with Apache 2. We will use PostgreSQL as the database management system. It is mainly based on the installation guide of Movim.

Prepare the database management system

After installing PostgreSQL on the system, we will prepare it. The goal is to create one user especially for the Movim web application which will have his own database and nothing else.

Create the `movim` user

First, connect yourself as the postgres user which will allow you to send requests to the PostgreSQL engine.

	su - postgres

Then you should be able to create the user. Either you enter the PostgreSQL engine to create the user (and you should find the right SQL command to do it), or you use the facility createuser. We will create a user with no rights to create database or users and with no superuser rights. We’ll put no password since we will connect from the localhost (see next sections for allowing connections from localhost). Adding the --echo option will allow you to see what should be the SQL commands to send.

	createuser --no-createdb --echo --encrypted --inherit --login --no-createrole --no-superuser movim

Create the `movim` database

For the Movim web application, a dedicated database, called movim will be created. We must associate it to the user movim and we will use the a unicode encoding (see PostgreSQL documentation). To create the database, you can use the createdb facility. The last field can be used to add a description.

	createdb --echo --encoding=Unicode --owner=movim movim "Movim social network database"

Giving correct rights

The database and the web application will be hosted on the same server. We don’t need a password for the web application to connect to the database; we will only trust any connection to the movim database system from movim user on the localhost (see PostgreSQL documentation).

The Debian documentation help to understand where to begin. We will edit the file /etc/postgresql/x.y/main/ph_hba.conf and add 2 lines (one for IPv4 and one for IPv6) to allow connection from localhost with movim user on movim database.

	# DO NOT DISABLE!
	# If you change this first entry you will need to make sure that the
	# database superuser can access the database using some other method.
	# Noninteractive access to all databases is required during automatic
	# maintenance (custom daily cronjobs, replication, and similar tasks).
	#
	# Database administrative login by Unix domain socket
	local   all             postgres                                peer

# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     peer
# IPv4 and IPv6 local connections for Movim
host    movim           movim           127.0.0.1/32            trust
host    movim           movim           ::1/128                 trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            md5
# IPv6 local connections:
host    all             all             ::1/128                 md5

Bonus

You may want to see what’s in the database. You should connect as the postgres user then run psql.

In the PostgreSQL shell, you may try \l to list all the databases. Then you may try other commands considering the documentation.

Installing Movim

You did the hard part, installing Movim is very easy. First, download it on the website. Then unzip it in the directory yuo want (don’t forget to create a entry with Apache).

Now, you need to configure the database for your Movim instance. In the unzipped archive, you’ll find a config/db.example.inc.php. Copy this file in config/db.inc.php then edit it. You should be able to modify it yourself without explanation (but I will help you, the following file).

	<?php
	# This is the database configuration of Movim
	# You need to copy an rename this file to 'db.inc.php' and complete the values
	$conf = array(
		# The type can be 'pgsql' or 'mysql'
		'type'        => 'pgsql',
		# The database username
		'username'    => 'movim',
		# The password
		'password'    => '',
		# Where can we find the database ?
		'host'        => 'localhost',
		# The port number, 3306 for MySQL and 5432 for PostGreSQL
		'port'        => 5432,
		# The database name
		'database'    => 'movim'
	);

Now, you should be able to access your website.

References

The official website for Movim
Installation guide for Movim
PostgreSQL documentation
More hints about PostgreSQL with Debian documentation

Generate slides from DocBook

2014-06-13T00:00:00+02:00

Introduction

I’m a big fan of LaTeX. However, when it comes to details, LaTeX is not what it sells you: you can’t really separate the content from the display when you need something nicer than the default. Moreover, the semantic description of parts of the content in LaTeX can be improved. But why? DocBook do the job, right?

Yes, DocBook seems to do a big part of the job. However, it is still difficult to produce documents from DocBook (FOP… seriously, you don’t want to use that, it’s heavy). You can still produce HTML/XHTML documents but for reports, you may want PDF. However, for slides, with a bit of CSS and JavaScript, it start to be interesting. Let’s do something like that!

In this article, we will parse a simple slide (with the DocBook and DocBook Slides formats) on a Linux, thanks to the xsltproc (part of the XSLT project of Gnome).

Get the DocBook stylesheets

To generate slides from DocBook format, you first need to get the stylesheets from the official forge docbook.sourceforge.net. You can download latest snapshots from snapshots menu or directly from snapshots.docbook.org.

Download the ns version (and the doc if you want) then unzip it somewhere. We’ll find everything we need for slides into docbook-xsl-ns-snapshot/slides/ directory.

How it works?

First you need to create your DocBook file (XML format). For slides, an extension is provided with another namespace which allow you to add a few more things related to a slides. It is called DocBook Slides and the namespace is http://docbook.org/ns/docbook-slides. Once your document is created, you need to parse it with an XSL stylesheet to produce your HTML/XHTML file. Then, there is a few other details to make it work.

Create the document

First of all, this is simple DocBook file. you can use either the DocBook roots or the specific root from DocBook Slides extension to begin your document. We will use the extension’s root for the XML file.

	<?xml version='1.0'?>
	<dbs:slides xmlns="http://docbook.org/ns/docbook"
		xmlns:dbs="http://docbook.org/ns/docbook-slides">
		...
	</dbs:slides>

Each slide will be enclosed in a <dbs:foil> element. For each <dbs:foil> element, you can add <title> then content (you can add a lot of things but we will make it simple for now).

Let’s try a simple slide with 3 slides with the file docbook-slide-example.xml.

<?xml version='1.0'?>
<dbs:slides xmlns="http://docbook.org/ns/docbook"
		xmlns:dbs="http://docbook.org/ns/docbook-slides"
		xmlns:xlink="http://www.w3.org/1999/xlink">
	<title>Create DocBook slides</title>
	<dbs:foil>
		<title>Introduction</title>
		<para>This is very simple, you'll see</para>
	</dbs:foil>
	<dbs:foil dbs:incremental="1">
		<title>Create itemized list</title>
		<itemizedlist>
			<listitem>
				<para>For help...</para>
			</listitem>
			<listitem>
				<para>... you can refer to...</para>
			</listitem>
			<listitem>
				<para>
					... the official website
					<link
						xlink:href="http://www.docbook.org/">
						DocBook
					</link>
				</para>
			</listitem>
		</itemizedlist>
	</dbs:foil>
	<dbs:foil>
		<title>Display source code</title>
		<para>Every XML file should start with the following line</para>
		<programlisting>
			&lt;?xml version='1.0'?&gt;
		</programlisting>
	</dbs:foil>
</dbs:slides>

Parse the document

Every stylesheet that you can use to generate slides are in docbook-xsl-ns-snapshot/slides/xhtml/. The one we are interested in to create fun presentation is slidy.xsl. The document you want to parse is docbook-slide-example.xml. To parse it, we will use xsltproc then redirect the output in our XHTML file.

	xsltproc \
		docbook-xsl-ns-snapshot/slides/xhtml/slidy.xsl \
		docbook-slide-example.xml > docbook-slide-example.xhtml

Now, you can display the file. However, you’ll see nothing spectacular since we do not furnish the CSS and JavaScript files.

A few more details

To complete the creation of our slides, we will need to copy the needed CSS and JavaScript files into the right directory. All of these files can be found into docbook-xsl-ns-snapshot/slides/slidy/. This slidy directory is expected to be found in a directory called files next to the XHTML file. Let’s copy the directory.

	mkdir files/
	cp -R docbook-xsl-ns-snapshot/slides/slidy/ files/slidy

In your directory, you should now see the following file structure.

	|
	|-- docbook-slide-example.xhtml
	|-- docbook-slide-example.xml
	|-- docbook-xsl-ns-snapshot
	|   |-- slides
	|   |   |-- ...
	|   |   +-- ...
	|   +-- ...
	+-- files
		+-- slidy
			|-- ...
			+-- ...

Reload the XHTML file in your browser, you should be able to see the nice slide template from the W3C!

References

docbook.sourceforge.net, the official forge of DocBook
snapshots.docbook.org, where you can download snapshots of DocBook stylesheets
In docbook-xsl-ns-snapshot/slides/doc/, the file slides.xml can be processed with xsltproc and slidy.xsl to produce a short documentation.

Connect Firefox OS device to Arch Linux

2014-03-19T00:00:00+01:00

Introduction

Firefox OS, what is that? If it’s your question, this article is probably not for you. You may take some information on Wikipedia.

Now, you know. So, Firefox OS (or Boot2Gecko or B2G) is developed in HTML5 (by HTML5, I mean HTML5+CSS3+ECMAScript/JS). This is a wonderful opportunity to develop new applications in a simple langage. However, to be able to do push applications on your new smartphone, you need to be able to connect it to your computer.

This article will explain how to connect a smartphone device (Geeksphone Revolution) running Firefox OS on an Arch Linux distribution. Since the connection is done through the Firefox browser, I will only consider version of the browser greater than 26.0 which allow the installation of the last Firefox OS simulator extension.

The steps

There is only a few steps to do. First of all, you will need to configure your smartphone to be able to connect it to the computer with a remote debugging. This article is only about a 1.3 version of Firefox OS, the steps may be different for an older version.

Then we will configure the computer. First of all, we need to configure the hardware part of the computer, then we will configure the software part.

The smartphone

For the smartphone, the configuration is pretty simple. You need to enable the remote debugging. Go into the Parameters menu and find Device Information > More Information > Developer. You should find a Remote debugging option. If it’s not enabled, check the checkbox.

You should also deactivate the screen locking in Parameters.

The computer

Hardware part

On the web, you’ll find a lot of website that tells you to install adb tool, even on the Mozilla website. However, it seems that this manipulation is not needed anymore with the last version of the debugging tools in Firefox browser. You can only install in your browser the ADB Helper extension. W’ll see this step in the next section.

However, you still need to configure udev to read your USB device. Edit a file into /etc/udev/rules.d/, for example 51-firefoxos.rules and put the following inside.

SUBSYSTEM=="usb", ATTR{idVendor}=="8087", ATTR{idProduct}=="0a16", MODE="0666"

You may check the idVendor of your device. First of all, plug it to your computer. Then use lsusb. If there is multiple lines, unplug it and type lsusb; one line should have disappear, it’s the one we are interested in. In the line, you should see something like the following.

Bus 001 Device 007: ID 8087:0a16 Intel Corp.

In the field 8087:0a16, 8087 is the idVendor. Save your rule and reload them with the following command (with root privileges).

udevadm control --reload-rules

The software part

First step in software part is installing last version of Firefox browser on your desktop (if it’s not already done)… Done? Ok, let’s continue.

You can now open the App Manager. You should find it under Tools > Web Developer > App Manager. At the bottom, you should find a Start Simulator button. Click on it. If a Firefox OS simulator is already installed, you can start it now and test a Firefox OS simulator. You may also add one (click on Add button and follow the instructions); but that’s not the subject of this article. We want to launch the remote debugging on the smartphone device.

When you clicked on the Add button above (if you didn’t, now you should!), you’ll see a link to install the simulators (a big green button with Install Simulator), at least at the day I’m writing this article. Click on it. It will propose you to install the simulators (but you already did it before) and an extensions called ADB Helper (remember, I already talked about it in the hardware part). Install it.

I think now you should see your connected device with a button next to the button to start the simulator. If not, try to restart your browser at least.

Possible errors

Get `????????????` in the `App Manager`

When you connect your smartphone device, the new button appearing in the App Manager you should display characters and numbers (for example, Clovertrail5B2040C1 in my case). However, at the beginning of my experimentations, it displayed ???????????? and I was unable to start the remote debugging.

If you have adb installed (see below), you also may try adb devices to see a similar result ???????????? no permissions. From what I understand, it’s mainly a authorization problem and depends only of the udev rule so trying to write correctly this rule should make your problem disappear. Looking for information about that on the web, you may find rules that adds (at the end of the line) OWNER or GROUP. Be careful to be the right user or in the right group if you use that.

Force reloading of `udev` rules

If the reloading of udev rules, you may try to force to reload your own rules with the following command (with root privileges).

udevadm trigger

Installing `adb` tools

I didn’t find a nice way to install adb tools. These are the 2 methods I found for Arch Linux.

Using `yaourt`

The simplest way is to install the package android-sdk-platform-tools. However, even if the package is highly ranked on AUR, I’m not really fond of installing package through yaourt.

Installing `adb` from Android website

You can download the Android Developer toolkit. This is an archive; uncompress it. However, they move the adb tool in this version of the toolkit (see tools/adb_has_moved.txt).

Considering the file ‘SDK README.txt’, you should launch tools/android from the root of the uncompress archive. You’ll see different packages you can install. Select Android SDK Platform-tools and install it.

If it’s not enough, you can also update the toolkit by running the following command from the root of the uncompress directory. However, it will download everything and more (around 10GB of data to download).

tools/android update sdk --no-ui

Now, you should have adb in platform-tools/adb.

To use it, try first this command when your smartphone device is plugged.

platform-tools/adb devices

If you have the error message ???????????? no permissions, then try the following (sometimes it works).

platform-tools/adb kill-server
platform-tools/adb devices

References

MDN is the first point of reference for everything concerning Firefox OS; for the hardware part, it will sometimes redirect you to Android documentation
28ADB.29”>Help to connect Android devices on the Arch Linux website

Screen backlight problem after suspend on Arch

2014-03-04T00:00:00+01:00

Introduction

On Arch, it seems that there are often problems with the screen backlight. For example, I had a problem when suspending my computer with an almost discharged battery. When I plugged my computer, it was fully discharged. Trying to restart it from suspend didn’t display something on my screen. Let me explain all the details of this problem.

Environment

First of all, let me describe the environment where it happens. I am using a ASUS N750JV computer, with Arch Linux installed on it. My graphic card is an Intel; this is the result of lspci for the Intel card.

VGA compatible controller: Intel Corporation 4th Gen Core Processor Integrated Graphics Controller (rev 06)

Description of the problem

I was using my computer on battery and reaching around 5% of battery charge, I decided to suspend my computer by closing it. I don’t really know what kind of program do the suspend by default on Arch Linux with Gnome.

However, before I can plugged my computer, the battery fully discharge. In an ideal case, the computer would have detected that it has only a few minutes to shutdown in a clean way. But I am not really sure that Arch (or Gnome, I don’t really know which component should be in charge of this) is doing it right.

When I tried to reboot the computer, the screen never shows me anything. After a few try to hard reboot (holding the power button then reboot), I resigned.

The computer is started

However, I discovered something interesting, the computer seems to start; I had the first initializations steps displayed. After these steps, the screen shutdown. Based on the assumption that the computer is started and that only the screen didn’t work, I tried to blindly connect my session then run a terminal then try to reboot. And it works!

Temporary solution

That’s a good news, because I can try to fix it, even if it’s blindly. I tried a lot of things I found on Internet to switch on backlight with no success.

The only final idea I had was: “I may try to restart properly the suspend procedure”. So instead of trying to swith on the screen backlight, I tried to suspend my computer in CLI because I suspected than my problem came from a bad restart after the suspend.

sudo systemctl suspend

The computer effectively suspended, the hard drive stopped, it works! Now, typing space to restart it… and… magic! The screen came back to life.

However, first thing I tried is to close my computer then open it to check if it works well. That’s a deception because, I’m back to my initial problem. Trying to reboot it: same problem. This solution is only a temporary solution to debug but the problem is still there.

The solution

Now, I’m pretty sure that it’s a only a problem of my screen backlight. I then looked the specific Arch wiki page about backlight. And the solution is in the brightness file.

There is a brightness file under /sys/class/backlight. This brightness only contains a number with the intensity of the brightness of the screen. I have three different subdirectories in the backlight directory so I checked the brightness file in each of them (acpi_video0, acpi_video1 and intel_backlight). And the one in acpi_video1 was containing the value 0.

At this stage, be careful because you can’t just put any value in this file without knowing what the maximum value. In the same folder than your brightness file, you should find the max_brightness file which contains the maximum allowed value. For example, for my acpi_video1 directory, it’s 100 but for my intel_backlight directory, it’s 5273. Now, you can replace the value in the brightness file.

echo 100 > /sys/class/backlight/acpi_video1/brightness

Bonus

Check you computer is started

That’s something I didn’t tested, because I was lacking the tools. There is two ways I learned.

The first one is easy, you just plug another screen on VGA/DVI/DisplayPort. If it works, it’s a proof that your computer is started. However, it doesn’t work, it doesn’t proove anything; it’s possibly one of the endless problem of connecting a second screen on Linux!

A more tricky one is using a light. The hypothesis is that your screen is working, the only thing that is not working is the backlight of the screen. In a black room, using the light with the right angle on your screen, you should see the LCD pixels displaying things.

References

Backlight page for Arch Linux

Transform portrait to squared image by filling

2014-03-04T00:00:00+01:00

Introduction

This is my problem. I have a photo (portrait.png).

From this photo, I need a squared image. However, taking the largest possible croping, this is what I can get.

Of course, I don’t want the face to be cropped like this. The following article is presented one possible solution to get something like this picture.

In this article, we will only use ImageMagick.

Explain the procedure

To create a squared version of the portrait, I will extend the original picture by adding borders on left and right. Of course, I don’t want simple white borders so I will fill these borders; to do this, I will blur the current border of the picture (mainly green) with a blurred effect.

However, if I keep the existing picture as it is and add fuzzy colors in the borders, I will see the boundary between the center part of the image (containing the face) and the borders (only made of blurred colors). To avoid this virtual border, we will create borders with an increasing blurred effect as we go from the center of the image to the left and right borders. Of course, not exactly from the center since we need to keep the very center of the image (the one which contains the face) not blurred.

To process this blur effect, we will use a mask which will give the intensity of the blur for each pixel of the image.

Creating a mask

The mask will be a black and white image, white telling us that we want the maximum blur effect for this pixel and black telling us that we do not want any blur on this pixel. Basically, we want black on the important part of the visage then a black to white gradient as we go to the left and right borders.

Determine the parameters of the gradient

We will build it step by step. First, we need to define the size of our image to define the size of the gradients. identify will do the job.

identify portrait.png

Which results in the following informations.

portrait.png PNG 1111x1400 1111x1400+0+0 8-bit sRGB 1.854MB 0.000u 0:00.000

We will make a 1400x1400 squared image. This means that we will add 289 pixels on the left/right of the image (the half of it, 144.5 pixels, for each side).

To add borders , we can use the following method.

Note

About the transparency

We will add transparent borders for the display but for the purpose of the mask, it is not required since all we be erased with black and white. All transparent parts in the image have been displayed as a checkerboard pattern in order to clearly identify it. The method is the following:

convert your_image.png \
	-size 1400x1400 \
	pattern:checkerboard \
	-compose dstover \
	-composite \
	your_image_with_checkerboard.png

convert portrait.png \
	-bordercolor none \
	-border 144.5x0 \
	portrait.borders.png

Then we need to define the limit where we can begin the gradient. I choose the 250 pixels on each side as in the following image.

Building a one side gradient

I can now begin to create the gradient. I will create the right gradient first.

convert \
	-size 1400x1400 \
	xc: \
	-sparse-color Barycentric '1150.0,0 black %w,0 white' \
	mask.gradient.right.png

Then for the left gradient, we can just rotate it.

convert \
	mask.gradient.right.png \
	-rotate 180 \
	mask.gradient.left.png

Then to obtain the final mask, you combine the two parts.

convert \
	mask.gradient.right.png \
	mask.gradient.left.png \
	-compose lighten \
	-composite \
	mask.png

Fill the image

We now have the mask, we will use it on our image to produce the desired result. Of course, we will apply it on the squared version of our image, the previous portrait.borders.png file. To blur the image considering the mask, we will use the blur option from -compose. This option can take an argument, the same kind of argument that the -blur option takes (the intensity of the blurred effect). In our case, the intensity should be enough high to reach the borders of the image. Remember, the blur effect is an operation that take, for each pixel, an average color of all the pixels around; what I mean by the intensity of the blur effect is the diameter (in number of pixel) of the circle where we do this average.

Let’s try with a arbitrary value: 24.

convert \
	portrait.borders.png \
	mask.png \
	-compose blur \
	-set option:compose:args 24 \
	-composite \
	portrait.blur24.png

It seems that it’s not sufficient, and it already took a long time to generate the image… Wait! We forget something. It’s an average of colors but we have only transparent colors on the borders. So there is probably pixels in this image that have a blurred colored corresponding to the image but with a majority of transparent color: we do not see it. To only keep the RGB color (which is the meaningfull information in our picture), we will just remove the transparency in the image (deactivate the alpha channel).

convert \
	portrait.blur24.png \
	-alpha off \
	portrait.blur24.rgb.png

There is clearly a problem. The size of our blurred effect was so small that the pixels on the extrems didn’t beneficied of any color. We need to increase the value of our argument. This time, we will apply the blurred effect and remove the transparency in one operation.

convert \
	portrait.borders.png \
	mask.png \
	-compose blur \
	-set option:compose:args 48 \
	-composite \
	-alpha on \
	portrait.squared.png

We finally have it, the squared portrait.

References

ImageMagick website, the only good reference to use ImageMagick

Display notifications with IRSSI

2014-01-17T00:00:00+01:00

Introduction

Since a few months now, I used irssi as my IRC client. I like it because it is highly configurable and because it’s CLI. However, one thing is a bit annoying about irssi: it has no notifications when somebody talk to me. Either it is some ping on a public chan or a private message, if I don’t check my terminal, I can’t know if somebody needs me.

However, another strength of irssi is that is can be extended with plugins. I will propose an extension to send notification through libnotify.

The plugin

To begin, I only used the plugin proposed by Thorsten Leemhuis called fnotify. On irssi, it is very easy to use a existing plugin. You need to put the Perl file fnotify.pl into your ~/.irssi/scripts/ directory. Note that it seems it must have the .pl extension to be loaded. Once it’s done, you can simply load the plugin into irssi doing the following command in an opened instance of irssi.

/script load fnotify

AUtomatically load the plugin

To load this plugin each time irssi is started, you must copy your plugin into ~/.irssi/scripts/autorun/. You may create a symbolic link too.

ln -s ~/.irssi/scripts/fnotify.pl ~/.irssi/scripts/autorun/fnotify.pl

Modifying the plugin to send notifications

The plugin is only catching the highlights and private message to print them into the file ~/.irssi/fnotify. To send notifications, we will use the libnotify library which provide the notify-send command. To call that command into our Perl script, we will use the system function.

To make it work, you will probably need to install libnotify (which may be there if you’re using Gnome) and Perl (which may be installed with irssi).

You can find the modify script here.

References

Based on fnotify script from Thorsten Leemhuis
Desktop notifications for irssi in screen through SSH in Gnome terminal
Library of irssi scripts

Patch a dependency of Maven

2014-01-13T00:00:00+01:00

Introduction

The other day, I was using a XWiki instance to make some crazy things (using XWiki in a way that is not the default). My point was to use the functionalities of creating subwikis. No specific configuration of XWiki after that point: Apache Tomcat v7.0.47, HSQLDB (embedded in the XWiki instance downloadable on the website) and XWiki v5.3. Is use Maven v3.1.1.

In the project, I have no build of XWiki, only a Maven dependency to xwiki-enterprise-web. My point is not to compile XWiki (which is a big software to compile), but only depend on its JAR files.

However, if I found a bug in XWiki, how can I manage that without compiling the entire XWiki software? And that is what I will talk about.

The problem

I have written some script to automatically call the WikiManager.CreateWiki page which will create a new subwiki based on a HTTP request with some parameters. Basically, I’m calling the WikiManager.CreateWiki with the URL /xwiki/bin/view/WikiManager/CreateWiki and the following parameters:

step=create
wikiname=mywiki
template=mytpl
wikiprettyname=My super wiki
ownerId=xwiki:XWiki.Admin
userScope=global_only
membershipType=open

However, when this page is called, it is not supposed that the subwiki already exists. First of all, it check is the subwiki already exists. Then it will also check that the subwiki is not in the process of creation; and this is where things may go wrong.

Checking the job status of the subwiki creation

To check the status of creation of a subwiki, we ask for the status of a job. The status of the subwiki creation can be called with this Velocity code.

#set($jobId = ["wiki", "provisioning", "wikiprovisioning.template", "mywiki"])
#set($percent = $services.wiki.template.getWikiProvisioningJobStatus($jobId).progress.offset)

$jobId is only a list of parameter where the last one is the identifiant of our wiki being created (see wikiname property above). Then we call the getWikiProvisioningJobStatus function from the script service WikiTemplateManagerScript.java.

Getting a bug

Getting a bug is usually the easy part! The bug is here, right under your nose. Look at the code of the script service. This is the code you should look for.

	public JobStatus getWikiProvisioningJobStatus(List<String> jobId)
	{
	    try {
	        return wikiTemplateManager.getWikiProvisioningJob(jobId).getStatus();
	    } catch (WikiTemplateManagerException e) {
	        return null;
	    }
	}

If the return of the getWikiProvisioningJob function is null, how can the function getWikiProvisioningJobStatus can return without exception? Everybody could have done this error, this is not the point. And since this is open source, the error as been reported in the JIRA of XWiki (XWIKI-9861) and fixed for the next release with the following code.

	public JobStatus getWikiProvisioningJobStatus(List<String> jobId)
	{
	    try {
	        WikiProvisioningJob wikiProvisioningJob = wikiTemplateManager.getWikiProvisioningJob(jobId);
	        if (wikiProvisioningJob == null) {
	            return null;
	        }
	        return wikiProvisioningJob.getStatus();
	    } catch (WikiTemplateManagerException e) {
	        return null;
	    }
	}

Fix the bug without compiling XWiki

I cheat a little because we will not compile the entire XWiki software. However, we still need to compile a small part of XWiki and replace it with the bug fix.

This is the general overview of what we will do to patch the bug:

Unpack the source corresponding to the JAR we want to modify
Patch these sources
Compile the modified source
Unpack the JAR we want to modify in another directory
Move the freshly compiled classes into the unpacked JAR directory
Pack the JAR
Exclude the bugged JAR from your main project
Include your patched JAR in the main project

Create a new Maven module

To begin, our patch JAR will be a new module of our project. Into the root of the project, we create a basic pom.xml file with mvn archetype:generate.

The bug come from the xwiki-platform-wiki-template-script module (artifactId in Maven with the groupId being org.xwiki.platform). We will named our new module with xwiki-platform-wiki-template-script-patched as artifactId.

For the example, the groupId of this module will be com.myxwiki. The main project will be called myxwiki in the same groupId.

You should have something like this at the beginning of your pom.xml.

	<parent>
		<groupId>com.myxwiki</groupId>
		<artifactId>myxwiki</artifactId>
		<version>1.0-SNAPSHOT</version>
	</parent>
	<groupId>com.myxwiki</groupId>
	<artifactId>xwiki-platform-wiki-template-script-patched</artifactId>
	<version>1.0-SNAPSHOT</version>
	<name>XWiki - Platform - Wiki - Template - Script - Patched</name>

Patch the sources

To patch the source, we will create a dependency to the java-source of the module we are looking for then use the maven-dependency-plugin (for unpacking) combined with maven-patch-plugin (for patching).

First of all, we will define directories where to unpack the sources.

	...
	<properties>
	    <src.directory>${project.build.directory}/xwiki-platform-wiki-template-script-src</src.directory>
	</properties>
	...

In the build part, we need to define where is the source directories.

	...
	<build>
	    <sourceDirectory>${src.directory}</sourceDirectory>
	    ...
	</build>
	...

We will now use 2 plugins. The first to unpack the sources and the second to patch them.

Unpack the sources

First of all, we use the maven-dependency-plugin to unpack the sources. We have already defined the sourceDirectory. We still need to give where are the sources.

	<plugin>
	    <groupId>org.apache.maven.plugins</groupId>
	    <artifactId>maven-dependency-plugin</artifactId>
	    <executions>
	        <execution>
	            <id>unpack</id>
	            <phase>generate-sources</phase>
	            <goals>
	            <goal>unpack</goal>
	            </goals>
	            <configuration>
	                <artifactItems>
	                    <artifactItem>
	                        <groupId>org.xwiki.platform</groupId>
	                        <artifactId>xwiki-platform-wiki-template-script</artifactId>
	                        <version>5.3</version>
	                        <type>java-source</type>
	                        <outputDirectory>${src.directory}</outputDirectory>
	                    </artifactItem>
	                </artifactItems>
	            </configuration>
	        </execution>
	    </executions>
	</plugin>

Create the patch

First of all, we need to create the patch. To create the patch, we first need to clone xwiki-platform (do not forget that you should get the v5.3 tagged version because this bug has been fixed in the master branch). Then, we will modify the source we want (WikiTemplateManagerScript.java). Once it is done, we can generate the patch using the following command from the java directory (xwiki-platform/xwiki-platform-core/xwiki-platform-wiki/xwiki-platform-wiki-template/xwiki-platform-wiki-template-script/src/main/java).

git diff -p --src-prefix '' --dst-prefix '' --relative -- org/xwiki/wiki/template/script/WikiTemplateManagerScript.java

You should get the following file.

	diff --git org/xwiki/wiki/template/script/WikiTemplateManagerScript.java org/xwiki/wiki/template/script/WikiTemplateManagerScript.java
	index 74ca25f..8e49bf9 100644
	--- org/xwiki/wiki/template/script/WikiTemplateManagerScript.java
	+++ org/xwiki/wiki/template/script/WikiTemplateManagerScript.java
	@@ -40,6 +40,7 @@
	import org.xwiki.wiki.descriptor.WikiDescriptor;
	import org.xwiki.wiki.descriptor.WikiDescriptorManager;
	import org.xwiki.wiki.manager.WikiManagerException;
	+import org.xwiki.wiki.provisioning.WikiProvisioningJob;
	import org.xwiki.wiki.template.WikiTemplateManager;
	import org.xwiki.wiki.template.WikiTemplateManagerException;
	@@ -223,7 +224,11 @@ public Exception getLastException()
	    public JobStatus getWikiProvisioningJobStatus(List<String> jobId)
	    {
	        try {
	-            return wikiTemplateManager.getWikiProvisioningJob(jobId).getStatus();
	+            WikiProvisioningJob wikiProvisioningJob = wikiTemplateManager.getWikiProvisioningJob(jobId);
	+            if (wikiProvisioningJob == null) {
	+                return null;
	+            }
	+            return wikiProvisioningJob .getStatus();
	        } catch (WikiTemplateManagerException e) {
	            return null;
	        }

Apply the patch onto the sources

In your new Maven module root, you now need to create the directory src/main/patches and put your patch (WikiTemplateManagerScript.patch) inside.

Now, we can add the plugin which will patch our source with the file we have just created.

	<plugin>
	    <groupId>org.apache.maven.plugins</groupId>
	    <artifactId>maven-patch-plugin</artifactId>
	    <version>1.1.1</version>
	    <configuration>
	        <skipApplication>false</skipApplication>
	    </configuration>
	    <executions>
	        <execution>
	            <configuration>
	                <patchDirectory>src/main/patches/</patchDirectory>
	                <targetDirectory>${src.directory}</targetDirectory>
	                <patches>
	                    <patch>WikiTemplateManagerScript.patch</patch>
	                </patches>
	                <patchTrackingFile>${project.build.directory}/patch-log.txt</patchTrackingFile>
	                <naturalOrderProcessing>true</naturalOrderProcessing>
	                <ignoreWhitespace>true</ignoreWhitespace>
	            </configuration>
	            <phase>process-sources</phase>
	            <goals>
	                <goal>apply</goal>
	            </goals>
	        </execution>
	    </executions>
	</plugin>

Create the JAR

To create our modified version of the JAR, we need to unpack the original version. The point of doing this operation is to keep existing files that we may not have in our sources (components.txt for example). First of all, we will unpack the JAR with the same plugin than the one for the source: we will add another <artifactItem> XML element.

	<artifactItem>
	    <groupId>org.xwiki.platform</groupId>
	    <artifactId>xwiki-platform-wiki-template-script</artifactId>
	    <version>5.3</version>
	    <type>jar</type>
	    <outputDirectory>${bin.directory}</outputDirectory>
	</artifactItem>

You can note that the unpacked files will be in the binary directory, exactly where the modified sources (that we patch), will be compiled. So we do not need anything else since the unpacking is done, then the compiling which will replace the compiled files (WikiTemplateManagerScript.class file is the only file). However, we need to make the JAR original project a dependency of our module.

	<dependency>
	    <groupId>org.xwiki.platform</groupId>
	    <artifactId>xwiki-platform-wiki-template-script</artifactId>
	    <version>5.3</version>
	    <type>jar</type>
	</dependency>

The new JAR!

Now, you just need to mvn clean install and see the result. You should find the JAR file xwiki-platform-wiki-template-script-patched-1.0-SNAPSHOT.jar into your target directory. Now, we need to use it in our main project.

Exclude the bugged JAR

To exclude the bugged file, we will use the maven-war-plugin plugin. In your pom.xml, you should have a dependency to XWiki somewhere. Something like that for example.

	<dependency>
	    <groupId>org.xwiki.enterprise</groupId>
	    <artifactId>xwiki-enterprise-web</artifactId>
	    <version>5.3</version>
	    <type>war</type>
	</dependency>

You can add the plugin which will exclude the xwiki-platform-wiki-template-script-5.3.jar JAR file.

	<plugin>
	    <groupId>org.apache.maven.plugins</groupId>
	    <artifactId>maven-war-plugin</artifactId>
	    <configuration>
	        <packagingExcludes>
	            WEB-INF/lib/xwiki-platform-wiki-template-script-5.3.jar
	        </packagingExcludes>
	    </configuration>
	</plugin>

Into the <packagingExcludes> element, you may use wildcard pattern * to find more than one file.

Include the patched JAR

To include you new patched JAR, you just need to add a dependency to it.

	<dependency>
	    <groupId>com.xwikisas</groupId>
	    <artifactId>eesc-xwiki-platform-wiki-template-script</artifactId>
	    <version>1.0-SNAPSHOT</version>
	</dependency>

References

Finally, the project I was working on to apply this patch is hosted by Github and is called EESC XWiki. You can find the patch module inside eesc-xwiki-platform-wiki-template-script and the main project for the WAR in eesc-xwiki-web/eesc-xwiki-web-base (which is not the root of our project).

Get information from a XWiki database

2013-12-12T00:00:00+01:00

Introduction

It is really a problem when in a file you have characters displayed as something like ^@ or <E5>. Indeed, yesterday, my XWiki (installed in local on my computer) has been crashed by a computer freeze. I hardly reboot my computer and never been able to start the wiki after that.

On this wiki, I was developping some new Velocity scripts and if I was not able to get them back, it would be a lot of lost work. I used the default database system for XWiki which is HSQLDB.

Step 1 - Backup the work

My first action has been to backup all my work before deploying a new XWiki. On this step, I was lucky because all my work was store in a subdirectory. All stuff concerning the database was in a xwiki.data directory and this is what this directory contains.

xwiki.data
|--> database/
|--> extension/
|--> jobs/
|--> lucene/
|--> solr/

Step 2 - The database

Exporting the database

The database file is inside de database/xwiki_db.lobs file. I copy it into a new directory where I will be able to parse it and recover my work.

For example, you can do this.

$ cd ~/
$ mkdir restore-xwiki/
$ cd restore-xwiki/
$ cp <path-of-xwiki-database>/xwiki_db.lobs xwiki.db

What’s in this database file?

This file is almost a simple text file with not-valid XML in it. Almost because there is a lot of ^@ in it; indeed, between each readable character, there is the ^@ character. Not-valid XML file because it is in fact a concatenation of multiple well-formed XML file (i.e. you have multiple <?xml …?> elements in the file).

Step 3 - Clean the file

Your file is now called xwiki.db. We will parse it to remove all the ^@ characters. You may want to try with any file editor like gedit, emacs or vim. However, the file is a database and in my case, was a more than 300MB file. Opening a large file like this may be a problem (and parsing it to replace characters and even bigger problem in this case).

To parse this file, we will use the tr command which is way more efficient. The only thing to do before using tr is to identify the ASCII/UTF-8/WHATEVER code of this ^@ character.

Identifying the character

To identify this character, I used a tricky way which consist to take a small part of this file, opening it with my favorite editor vim and identifying the code under it.

To create the small part of the database file, use the following then open with vim.

$ cat xwiki.db | head > temp.db
$ vim temp.db

Once you’re inside vim, you can see ^@ characters. Place your cursor on it and type ga on your keyboard (ga is for Get ASCII). You should see at the bottom of the window the code of the character we are looking for: this is the one with the code 0.

Bonus to identify the character

We could have done a easier method to identify the character (but this method may not work on other cases). You can open vim and typing the following: CTRL+v then CTRL+@. You should obtain the character we are looking for. Then to identify it, you can still use the ga shortcut.

Clean the file

Now, we will use the tr command which can replace each character of a file with another (this functionality doesn’t interests us) and also delete specific characters from a file (this functionality seems more interesting!).

tr take the standard input as an entry and we will save the output in the xwiki.xml file (which is not a real XML file).

$ tr -d '\000' < xwiki.db > xwiki.xml

Step 4 - Find the XWiki document

Each page of the XWiki is stored in xwiki.xml as an XML file which is presented like that.

<?xml version="1.0" encoding="UTF-8"?>
<xwikidoc>
<web>Main</web>
<name>WebHome</name>
[...]
<contentUpdateDate>1383876486000</contentUpdateDate>
[...]
<content>[...]</content>
</xwikidoc>

<xwikidoc> is the root element of these XML file. After this element, the <web> and <name> elements respectively describes the space and name of your XWiki page. After <name> element, there is a few header elements that describes the page. For example, you may find a few <object> elements that describes the objects stored in that page. One element that is interesting for us is the <contentUpdateDate> which contains the update date of this page. Finally, the <content> element contains the content of our page (in my case, the Velocity I am looking for, remember, I’m trying to restore it because I lost Velocity work).

Given these informations, it seems pretty easy to find only the document we want and copy/paste the content to save it. However, XWiki store each saved version of each page. Though if I’m looking for the page Main.WebHome, I will find multiple instances and that’s where the <contentUpdateDate> will help us.

Step 4.1 - Search a multiline pattern

The first thing we want to do is identifying the right page. And because a page is identified by it’s name and it’s space, we need to look into the file for a multiline pattern that contains <web> and <name>.

For a multiline research, people says (see references) you should be able to use the -P option of grep. But for a mysterious reason, the -P option doesn’t works for me and ends up with the following message.

[1]    2618 abort (core dumped)  /usr/bin/grep [...]

So I used another solution proposed in the same forum reference: pcregrep.

$ pcregrep -M -n -A 11 '<web>Main<\/web>[^<]*\n[^<]*<name>WebHome<\/name>' temp.xml

I add also the -n which numerotates each line: this will help us to find the line number of what we are looking for. And finally, I add the -A 11 option in order to display 11 lines after each matching pattern. Why I display 11 lines? That’s because 11 lines later, we can find the <contentUpdateDate> element which is the information we are interested in.

Step 4.2 - Filter the `<contentUpdateDate>` elements

With the previous command, we have searched for documents with a particular name and space. We have also displayed each pattern with some extra lines wich contains the following <contentUpdateDate> element. So each <contentUpdateDate> element that is displayed in this output is an potentially interesting one. Everything else in the output is of no interest now.

$ <output-4.1> | grep '<contentUpdateDate>'

Note	About the output In the following commands, I will use the `<output-4.?>` notation to designate the output of the previous step command.

Step 4.3 - Keep the numbers we want

In this step, I will remove the XML elements and keep only the line number and the date (the data inside the <contentUpdateDate> element). I will also place the date at the beginning of the line in order to be able to sort them in the next step.

$ <output-4.2> | sed 's/^\([0-9]\+\).*<contentUpdateDate>\([0-9]\+\)<\/.*$/\2-\1/g'

Step 4.4 - Sort by date and keep the last one

This is pretty easy since the sort command exists. To keep only the last one, we can use the tail command.

$ <output-4.3> | sort | tail -n 1

Step 4.5 - Get the line number

Now, we just need to get the line number and save it in a variable. We will use the awk command (we could also use the sed command) to get the second field of our line which should look like 1383876486000-2356 (the first field is the date and the second the line number).

$ LINE_NUMBER=`<output-4.4> | awk -F '-' '{print $2}'`

Step 4.6 - Find the end of the XWiki document

Now that we have the line number of the <contentUpdateDate> of the document we are looking for, we will parse the file from this element to find the first </xwikidoc> element (which is the close element of the document).

First of all, we will use grep -n '' in order to add the line numbers to our input. Then we will remove all the lines before our LINE_NUMBER with tail because we want the first </xwikidoc> after this line number. We will filter all the </xwikidoc> elements with grep and keep only the first with head. Finally, we will get the line number from this input with awk and store it in the END_LINE variable.

END_LINE=`cat xwiki.xml | grep -n '' | tail -n +$LINE | grep '<\/xwikidoc>' | head -n 1 | awk -F ':' '{print $1}'`

Step 4.7 - Find the start of the XWiki document

The operation is nearly the same that for the step 4.6. We will just revert some of the sub-steps. In this case, we will only keep the part of the document that is before LINE_NUMBER with head. Then we will look for <?xml pattern, keep the last found pattern and get then store the line number into the BEGIN_LINE variable.

$ BEGIN_LINE=`cat xwiki.xml | grep -n '' | head -n $(($END_LINE - 1)) | grep '<?xml' | tail -n 1 | awk -F ':' '{print $1}'`

Step 4.8 - Get the XWiki document

We now have BEGIN_LINE and END_LINE, we will be able to keep only this part of the database with head and tail with a bit of bash arithmetic.

$ cat xwiki.xml | tail -n +$BEGIN_LINE | head -n $(($END_LINE - BEGIN_LINE + 1 )) > Main.WebHome.xml

You can now get what’s in the <content> element. It is also possible to get the informations from the <object> (where you may also have code and configuration). If you want, you probably can create a XAR file from this XML file that you can import into your new XWiki.

You still may need to be careful about the encoding of the file because I notice that the encoding was latin1 in my case but a copy/paste of the content part into a new XWiki page need some utf-8 encoding.

References

Stress test XWiki for subwikis creations

2013-12-12T00:00:00+01:00

Introduction

I need to know how many subwikis can XWiki support without freezing or crashing. I prepared some kind of stress test (note that I’m not familiar with these kind of things) in order to massively create subwikis into a XWiki instance. By massively, I mean 1000 which seems to be sufficient (you will see that in the results).

My XWiki instance (5.3-milestone-2) is a local instance, on the 8080 port, runned on Jetty with HSQLDB.

Since my first method doesn’t work pretty well, I tried a second way to do this stress test but which may have a bias of measure.

First method - Velocity script called by a `curl` request

The Velocity script is using the wiki service which is new in the 5.3 version of XWiki. With this service, you can do a lot of interesting things:

exists(String) - Test the existence of subwiki
getById(String) - To get the wiki descriptor
createWiki(String, String, String, boolean) - Create a new wiki

This is the Velocity code I used and stored into WikiStressTestCode/FromRequest. The page take a HTTP request in entry with two parameters: * wikiname - the name of the created wiki * wikinumber - the number of the created wiki (useful for a stress test)

{{velocity}}
#set($wikiName = $!request.getParameter('wikiname'))
#set($wikiNumber = $!request.getParameter('wikinumber'))
#set($wikiId = "$!wikiName$!wikiNumber")
#if($wikiId.isEmpty())
#set($wikiName = 'Null')
#set($wikiNumber = 0)
#set($wikiId = 'null')
#end
#if($services.wiki.exists($wikiId))
#set($wiki = $services.wiki.getById($wikiId))
#else
#set($wiki = $services.wiki.createWiki($wikiId, $wikiId, 'xwiki:XWiki.Admin', true))
#end
#if($wiki)
#set($wikiPrettyName = "Wiki $wikiName #$wikiNumber ")
#set($discard = $wiki.setPrettyName($wikiPrettyName))
#set($wikiDescription = "Wiki $wikiName #$wikiNumber created for the stress test")
#set($discard = $wiki.setDescription($wikiDescription))
#set($discard = $services.wiki.saveDescriptor($wiki))
#else
#set($discard = $response.sendError(404, 'Error in creating the wiki'))
#end
{{/velocity}}

After this page has been created, you can call the page with a curl request as following;

#!/bin/bash

NUMBER=1
NAME=wiki
if [ $# -ge 1 ]
then
	NUMBER=$1
fi
if [ $# -ge 2 ]
then
	NAME=$2
fi

curl \
	--silent \
	--show-error \
	--user Admin:admin \
	-F "wikiname="$NAME \
	-F "wikinumber="$NUMBER \
	-o data/$NAME.html \
	http://localhost:8080/xwiki/bin/view/WikiStressTestCode/FromRequest

This file only create one request and take the number and the name as optional parameters.

I call the create-wiki.sh script into a loop from a second script file stress-test.sh. This second script file will measure the time of each request in order to observe the evolution of the time to create a new subwiki.

#!/bin/bash

TEST_NUMBER=3
NAME=test
LOG_FILE=stress-test-request.log
if [ -f $LOG_FILE ]
then
	rm $LOG_FILE
fi
if [ $# -ge 1 ]
then
	TEST_NUMBER=$1
fi

for num in `seq 1 $TEST_NUMBER`
do
	echo -n "Creating wiki "$Name" #$num..."
	/usr/bin/time --format=%e --output=$LOG_FILE --append ./create-wiki.sh $num $NAME
	TIME=`tail -n 1 $LOG_FILE`
	echo " ("$TIME"s)"
done

And now, this is the result of this first method.

As you can see, around 250 creations, XWiki seems to not respond. And I can check on XWiki that the first 250 subwikis have been created but not the others. I suspect the indexation of either Lucene or Solr to be too much loaded and freezing the whole wiki.

Second method - Groovy script

The Groovy script in WikiStressTestCode.InsideGroovy is merely doing the same as the Velocity script of the first method. However, it is not responding to a request but creating is own loop for the creation of the wikis. The problem of this method is that as long as the page is processing the iteration, the page is loading and there is no possibility to know the state of the loop (like “How many iteration there is?”).

This tricky part is done by writting the processing to a second page called WikiStressTestCode.InsideGroovyResults.

The second problem is to measure the time of execution. This is done in Groovy by using System.currentTimeMillis().

{{groovy}}
String wikiName = "test";
int testNumber = 1000;
resultsPage = xwiki.getDocument("WikiStressTestCode", "InsideGroovyResults");
resultsPage.setContent("");
resultsPage.save();
int wikiNumber = 1;
while(wikiNumber <= testNumber) {
	begin = System.currentTimeMillis();
	String wikiId = String.format("%s%s", wikiName, wikiNumber);
	if(services.wiki.exists(wikiId)) {
	wiki = services.wiki.getById(wikiId);
	} else {
	wiki = services.wiki.createWiki(wikiId, wikiId, "xwiki:XWiki.Admin", true);
	}
	if(wiki) {
	String wikiPrettyName = String.format("Wiki %s #%s", wikiName, wikiNumber);
	wiki.setPrettyName(wikiPrettyName);
	String wikiDescription = String.format("Wiki %s #%s created for the stress test", wikiName, wikiNumber);
	wiki.setDescription(wikiDescription);
	services.wiki.saveDescriptor(wiki);
	}
	wikiNumber = wikiNumber + 1;
	end = System.currentTimeMillis();
	String time = String.format("%d", end-begin);
	resultsPageContent = resultsPage.getContent();
	resultsPage.setContent(String.format("%s\n%s", resultsPageContent, time));
	resultsPage.save();
}
{{/groovy}}

To create an image, I can now load the page WikiStressTestCode.InsideGroovyResults to get all measured times.

And now, this is the result of this second method.

As you can see, I was not able to create my 1000 wikis. Almost 700 and crashed. First of all, the evolution seems to be as an exponential law, which is a bad news! However, we could level this conclusion given the fact that all subwikis are created in the same page: it is possible that the context of execution is growing (in memory) with a consequence on the subwikis creation. To simplify, the exponential law may be the consequence of my stress test and not from XWiki (or maybe both!).

Conclusion

You can find the XWiki code that has been used and scripts here:

On a server, you will probably be able to create around 1000 subwikis but the basic stress tests that I have made show that improvements should be done on XWiki before going further. Another method that I have not tested and would probably be the best for stress testing is to use JMeter.

Using `sed` to add new lines

2013-11-30T00:00:00+01:00

Introduction

sed is a powerful tool to edit files line by line. However, when you try to manipulate more than one line at a time, it becomes more difficult to create sed commands.

The point of this article is to find a pattern in a file and, when this pattern is found, add a new line just below.

The use case

Imagine that we have a lot of AsciiDoc documents. These articles have metadata at the beginning. However, the language is not defined, in any article. We want to add a metadata to define the lang of all articles (we know they all are in english).

For example, this is the metadata of this article

= Using `sed` to add new lines
:author: woshilapin
:email: woshilapin@gmail.com
:date: 30-11-2013
:slug: using-sed-to-add-new-lines
:category: tech
:tags: sed

We want to add the :lang: metadata after the :date: metadata.

The solution

With the sed command, we will use the a\ function to add a new line after the pattern we are looking for.

$ sed '/:date:/ a\
	:lang: en\
	' on-fichier.asciidoc

Notice that there is two new lines. The first says add a new line after the pattern you’re looking for'' and the second saysadd a new line at the end of the line I’m creating”. You may use the -i option of sed to replace in the file (the default is to print on the standard output).

References

Add new line after finding text

Problem with suspend on Linux

2013-11-22T00:00:00+01:00

Introduction

The suspend can be achieved with the power button (default configuration on GNOME 3) or with the Fn+F1 shortcut. After a fresh installation of Arch Linux, it worked but since few days, it does not resume well.

The symptoms: the computer (ASUS N750JV) is suspend and when resume, freeze with a screen showing some information about resume. Nothing to do except holding the power button for a hard shutdown.

Manage the problem

It seems that there is some hardware problems on resume. The best way to manage that is to blacklist these hardware. When looking at the freezing screen, the first problem is given by the the nouveau driver. In order to blacklist the nouveau driver, we will add the following line into the /etc/modprobe.d/blacklist.conf file. You can use any file into this directory; as long as it its name ends with .conf, it will be parsed.

blacklist nouveau

After after a reboot, you should now be able to suspend your computer and resume it without a freeze!

Fix a problem with fonts in web browser

2013-10-29T00:00:00+01:00

The problem

In your browser, you may have problems on some websites displaying strange fonts. For example, you may have seen the following fonts on Github.

This problem is due to bitmap font.

Solution

To deactivate bitmap fonts, you can use existing font configuration files you can find into /etc/fonts/conf.avail. To avoid bitmap fonts, you just need to link the 70-no-bitmaps.conf into /etc/fonts/conf.d directory.

$ cd /etc/fonts
$ ln -s conf.avail/70-no-bitmaps.conf conf.d/

Now, you can restart your X server for this new configuration taking effect.

References

Arch font configuration (tip on bitmap fonts)

In the rabbit hole - Presentation

2013-10-21T00:00:00+02:00

Introduction

This is the first post of a new blog. This blog will mainly talk about computer s and programmation.

The blog

Which system?

To run this blog, I used Pelican which is very simple to use. With this blog system, I can write my articles in different markdown syntaxes (Markdown, reStructuredText or AsciiDoc) and publish it as a static website; indeed, it creates the HTML files and publish them on the server. Easy to install, easy to use, easy to backup and easy to begin ;-)

Installing printer on GNOME 3

2013-08-11T00:00:00+02:00

Introduction

Installing a printer on an Arch Linux with GNOME 3 seems to be a little bugged. The printer is a HP LaserJet M1536 on an intranet (not a direct plugging).

Installing the service for printers

What else than CUPS? You can install the cups package which is the main reference about printing service.

$ pacman -S cups

Installing the driver

You may also install the driver for the printer. In the case of a HP LaserJet 1536, the driver is installed with hplip package.

$ pacman -S hplip

Starting CUPS service

You can start the CUPS daemon with systemd.

$ systemctl start cupsd

However, this will only start the daemon for the current session. In order to start it at every boot, use the following command.

$ systemctl enable cups

Adding the printer

This is where the problem begins! You may try to add the printer through the GNOME interface. Go in the preference menu, then the icon about printers. Click on add a printer. If the printer is on your intranet, you should see it after a small time of loading. Click on it to add the printer. You may have an error on this step. Why this is not working? No idea.

However, the printer can also be added through the CUPS interface. Open your web browser and go to your local port 631 (localhost:631/admin). You will be asked for a login and a password. These are the ones about the root account of your computer. Now, you can add the printer, it should work.

Possible errors

The printer does not appear

It seems that GNOME 3 has problems communicating with CUPS. In fact, the printers does not appear into the GNOME PDF viewer. You may try with Evince, which is a nice PDF viewer. Evince had no problem about seeing the installed printers.

References

Installing Skype on Arch Linux 64 bits

2013-08-11T00:00:00+02:00

Introduction

Skype is a mess since there is always problem using it on Linux. However, it’s probably one of the best VoIP software and at least, one of the most used, especially in business communications.

The installation proposed in this article is to install Skype on a 64 bits Arch Linux. One of the main problem, is the “64 bits” because Skype is still a 32 bits software.

Installing Skype

Because Skype is a 32 bits software, you need to configure pacman to be able to install 32 bits software through the multilib repository. To do that, uncomment the following lines into /etc/pacman.conf.

[multilib]
Include = /etc/pacman.d/mirrorlist

Now, run some update of pacman to be able to install Skype.

$ pacman -Syyu
$ pacman -S skype

The problem

You should now be able to run Skype and connect to your account. Moreover, you should be able to chat with your contacts with no problem. However, if you try to call someone or test your audio configuration, it will just hang up. Your contact will never see this call.

This problem is an audio problem.

Installing audio stuff

I will not enter in the details of ALSA, PulseAudio or other stuff because I don’t understanding a thing about these softwares. However, you should be able to make call with Skype installing two more 32 bits packages: lib32-libpulse and lib32-alsa-plugins.

$ pacman -S lib32-libpulse lib32-alsa-plugins

Now, you should be able to make your “Echo / Sound Test Service” or call your best friend to tell hom this story!

Possible errors

Microphone doesn’t work

Don’t worry, it may be nothing. Go to the GNOME preferences, inside Sound'' configuration. Now choose theInput” tab. You need to activate the microphone by switching the “On/Off” button on the right. You also may increase the volume to your convenience.

References

Install Eclipse on GNOME 3

2013-05-11T00:00:00+02:00

Introduction

I will install Eclipse and add an entry to GNOME to make Eclipse an application. I use the “Eclipse IDE for Java EE Developpers” version and I’m currently using GNOME 3.10.

Download and install Eclipse

To download Eclipse, you can go to http://eclipse.org/downloads/. Once it’s downloaded, you can extract it to /opt/. You should have the binary of Eclipse in /opt/eclipse/eclipse. You may extract Eclipse into /opt/eclipse-4.3.0 and create a symbolic link; this way, you can easily install a new version of eclipse by just modifying the link.

$ cd /opt
$ ln -s eclipse-4.3.0/ eclipse

Download a SVG icon for Eclipse

In order to make a cute icon in the GNOME launcher, we can also download an icon (in SVG format) for Eclipse.

$ cd /opt/eclipse/
$ curl -o /usr/share/pixmaps/eclipse.svg https://projects.archlinux.org/svntogit/packages.git/plain/trunk/eclipse.svg\?h\=packages/eclipse

Create the launcher for GNOME

To create the launcher, you need to create the following file /usr/share/applications/eclipse.desktop.

[Desktop Entry]
Encoding=UTF-8
Version=4.3.0
Type=Application
NoDisplay=false
Exec=/opt/eclipse/eclipse
Name=Eclipse
Comment=Launcher for Eclipse
Icon=eclipse

This is it. You can now try it!

References

Create a Custom Application Launcher on GNOME3

Install XWiki on Arch Linux

2013-05-11T00:00:00+02:00

Introduction

In this article, I detail the installation of Java and XWiki on an freshly installed Arch Linux. I will use the open source implementation of Java, OpenJDK and the generic installer of XWiki 5.2.

Install Java

On Arch Linux with OpenJDK, you need to install the jre7-openjdk and jdk7-openjdk packages.

$ pacman -S jre7-openjdk jdk7-openjdk

Install XWiki

You first need to download the generic installed on xwiki.org.

Once it is done, you can launch the installer with root privileges.

$ java -jar xwiki-enterprise-installer-generic-5.2-standard.jar

You can now follow the instruction. I choose to install XWiki into /opt/xwiki-5.2. If it is your first installation, you should install the Default Wiki because it will create a bunch of useful wiki pages to start.

When installation is finished, the installer asks for icon creation. I disable all icon creation because we do not really need them.

To finish this installation, I created a symbolic link on the directory of XWiki installation.

$ cd /opt
$ ln -s xwiki-5.2/ xwiki

Launch XWiki

To launch XWiki, you have a start script into the freshly installed directory.

$ cd /opt/xwiki
$ sh start_xwiki.sh

You should now be able to access your installed XWiki on http://localhost:8080 and enjoy!

Bonus

Log in

To log in this newly installed XWiki, you may use the default account:

login: “Admin”
password: “admin”

Applications for developers

If you want to develop some new applications for XWiki, you may install the following applications:

Collection Application: Makes easier the creation of a XAR file
Syntax Highlighting Application: Syntax highlighting for source, even for Velocity
AutoCompletion Application: Autocompletion in Velocity code

To install them, you need to be logged in with the Admin'' accound. Then, click onMore applications” in the panel on the right. You will see “Install new applications”; you can click on it.

Then search for each extension and click the Install'' button on the right then theContinue” button.

Reveal hidden pages

By default, XWiki hide some pages, mainly pages that are for developers and should not be displayed to a lambda user when using the search field (on the top-right). However, this could difficult to find some useful pages for a developer.

To make these page visible, you need to be logged in with the Admin'' account. Then go in thePreferences” in the profile (right in the top-bar). Click on the edit button (the little pen of the top right of the gray frame). You will have the possibility to change the property DISPLAY HIDDEN DOCUMENTS'' toYes”.

You may also change the DEFAULT EDITOR TO USE'' to Text” if you do not want WYSIWYG edition by default.

References

Installing the Standalone Distribution

Configure X server

2013-04-11T00:00:00+02:00

Introduction

To configure the X server on an fresh install of Arch Linux, you need:

Install the graphic driver
Remove nouveau or any nvidia driver
Install a desktop environment (GNOME in this case)
Configure the trackpad

Install the graphic driver

In this article, I will not detail the procedure to install a graphic driver for any ATI or NVidia graphic card. I will only focus on the basic graphic card of the motherboard which, in my case, is an Intel one.

To install the basic graphic driver for this Intel graphic card, you need to install the package xf86-video-intel.

$ pacman -S xf86-video-intel

And that’s it.

Removing other graphic drivers

To make sure your Intel graphic driver will be chosen, remove all other graphic driver that may have been installed during the default installation.

$ pacman -R nvidia xf86-video-nouveau

Install the desktop environment

To install the desktop environment (GNOME in this case), you just need to install the gnome package.

$ pacman -S gnome

By default, it will also install GDM, the GNOME Display Manager. In order to start it, we need a little more configuration. First, we activate GDM in the systemd system.

$ systemctl enable gdm.service

Secondly, you may have a non-US keyboard. However, GDM will take a US keyboard map by default. To change this, you can edit a X11 configuration file. For example, you can edit /etc/X11/xorg.conf.d/10-evdev.conf and change the following lines

Section "InputClass"
	Identifier "evdev keyboard catchall"
	MatchIsKeyboard "on"
	MatchDevicePath "/dev/input/event*"
	Driver "evdev"
EndSection

into

Section "InputClass"
	Identifier "evdev keyboard catchall"
	MatchIsKeyboard "on"
	MatchDevicePath "/dev/input/event*"
	Driver "evdev"
	Option "XkbLayout" "fr"
EndSection

Configure the trackpad

Your trackpad may not respond with the previous installation. It is probably because you miss the xf86-input-synaptics package. You can check if it is installed.

$ pacman -Qs xf86-input-synaptics

If it is not, you probably want to install it.

$ pacman -S xf86-input-synaptics

Possible errors

`nouveau` is still ativated

To check if nouveau is activated, use the lsmod command.

$ lsmod | grep 'nouveau'

If you find nouveau somewhere and that you’re desktop environment still doesn’t start, you may try to put nouveau on a blacklist.

$ echo 'blacklist nouveau' >> /etc/modprobe.d/blacklist.conf

Install Arch Linux on UEFI/GPT computer with BTRFS support

2013-02-11T00:00:00+01:00

Introduction

The goal of this article is to explain the installation of Linux distribution with some specificities:

The Linux distribution is an Arch Linux
The computer is UEFI compatible
The disk is a SSD
BTRFS will be the file system used
Use the BTRFS capabilities to split the /, /opt, /home and /var
The network is a wireless connection, configured with netctl

Run the Arch Linux installer

First of all, you need to download the ISO (around 550Mb) from the Arch Linux site (magnet or torrent) and copy it on a USB key or burn it on a CD. To make a copy on a USB key, you can use the dd tool.

$ dd if=archlinux-2013.10.01-dual.iso of=/dev/sdb

considering /dev/sdb is the device’s file for the USB key.

On Mac OS X, you will not have /dev/sdb but something like /dev/rdisk2.

You can now run your Arch Linux on the computer you want to install.

Initializations

First of all, you may want to change the keyboard layouts and the locales. For the keyboard layout, use the loadkeys tool.

$ loadkeys fr-pc

For the locales, edit the /etc/locale.gen file and uncomment the locale your want (you can uncomment more than one). Then generate the new locales and export your new default locale.

$ locale-gen
$ echo LANG=fr_FR.UTF-8 > /etc/locale.conf
$ export LANG=fr_FR.UTF-8

You may change the terminal’s font with the following.

$ setfont Lat2-Terminus16

Partitionning

Our installation will consider that the computer is UEFI compatible. UEFI is compatible with both MBR and GPT partitionning. However, GPT is the standard way to do it. In order to partition the disk, we will use the GPT version of fdisk, which is gdisk, on the main disk /dev/sda.

$ gdisk /dev/sda

gdisk brings you to a new shell. We will create 2 partitions:

The first partition will be the EFI partition
The second partition will be the partition for the Linux system

These are the operations you should do:

type o and validate to create the GPT partitionning
type n and validate to create a new partition
enter the partition number or just validate for the default (the default should be good)
enter the first sector or just validate for the default (the default should be good)
enter the last sector or the size of your volume; for EFI, enter +512M and validate
enter the type of the partition; for EFI, enter EF00 and validate
type n and validate to create a new partition
enter the partition number or just validate for the default (the default should be good)
enter the first sector or just validate for the default (the default should be good)
enter the last sector or the size of your volume; for Linux system, validate the default which is the last sector of the disk
enter the type of the partition; for Linux system, enter 8300 and validate
type w and validate to write all the modifications on the disk.

Formatting

The formatting is pretty easy. The EFI partition should be format in a FAT filesystem (16 or 32). The second partition will be format in BTRFS since it one of the purpose of this article.

$ mkfs.vfat -F32 -n "EFI" /dev/sda1
$ mkfs.btrfs -L "Arch Linux" /dev/sda2

Managing the BTRFS subvolumes

Mounting the main BTRFS volume

First of all, we will mount the BTRFS partition.

$ mkdir -p /mnt/btrfs-root
$ mount -o defaults,relatime,ssd,discard,space_cache /dev/sda2 /mnt/btrfs-root

discard and ssd mount options are optimizations for SSD drives. discard is to activate `Discard/TRIM'' optimizations and `ssd is for specific BTRFS optimization on SSD drives.

Creating and mounting the BTRFS subvolumes

Then we can create the subvolumes in the BTRFS partition.

$ mkdir -p /mnt/btrfs-root/__snapshot
$ mkdir -p /mnt/btrfs-root/__active
$ btrfs subvolume create /mnt/btrfs-root/__active/ROOT
$ btrfs subvolume create /mnt/btrfs-root/__active/home
$ btrfs subvolume create /mnt/btrfs-root/__active/opt
$ btrfs subvolume create /mnt/btrfs-root/__active/var

We can now mount the subvolumes in the right order.

$ mkdir -p /mnt/btrfs-active
$ mount -o \
	defaults,nodev,relatime,ssd,discard,space_cache,subvol=__active/ROOT \
	/dev/sda2 /mnt/btrfs-active

We create the directories in this new-mounted subvolume and mount the other subvolumes.

$ mkdir -p /mnt/btrfs-active/home
$ mkdir -p /mnt/btrfs-active/opt
$ mkdir -p /mnt/btrfs-active/var/lib
$ mount -o \
	defaults,nosuid,nodev,relatime,ssd,discard,subvol=__active/home \
	/dev/sda2 /mnt/btrfs-active/home
$ mount -o \
	defaults,nosuid,nodev,relatime,ssd,discard,subvol=__active/opt \
	/dev/sda2 /mnt/btrfs-active/opt
$ mount -o \
	defaults,nosuid,nodev,noexec,relatime,ssd,discard,subvol=__active/var \
	/dev/sda2 /mnt/btrfs-active/var

Tricky part for the `/var/lib` directory

However, the /var/lib is the directory in which some interesting files are installed and we want to keep them in the __active/ROOT subvolume to make easier the snapshot procedure. Remember, we create a directory in this subvolume. So we will bind it.

$ mkdir -p /mnt/btrfs-active/var/lib
$ mount --bind /mnt/btrfs-root/__active/ROOT/var/lib \
	/mnt/btrfs-active/var/lib

Mount the EFI partition into the main directory layout

At this point, we should mount the EFI partition as the /boot directory.

$ mkdir -p /mnt/btrfs-active/boot
$ mount -o \
	defaults,nosuid,nodev,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro,discard \
	/dev/sda1 /mnt/btrfs-active/boot

Install the base system

Before being able to install the base system, you need to configure a network connection. If you want a wireless connection, you can simply use wifi-menu. It will display you a menu where you can choose a wireless network and enter your password (in clear).

You may update the list of mirrors /etc/pacman.d/mirrorlist before running the following command.

The base package will install a base system for your Arch Linux. Because we use a BTRFS system, we also install btrfs-progs package (we could install it later too).

$ pacstrap /mnt/btrfs-active base btrfs-progs

We must now generate and configure the fstab file with the help of genfstab.

$ genfstab -U -p /mnt/btrfs-active >> /mnt/btrfs-active/etc/fstab

You should modify the /etc/fstab file and make it looks like something like the following.

tmpfs					/tmp		tmpfs	rw,nodev,nosuid 0 0
tmpfs					/dev/shm	tmpfs	rw,nodev,nosuid,noexec 0 0

# /dev/sda1 LABEL=Arch\x20Linux
UUID=...				/		btrfs	rw,nodev,relatime,ssd,discard,space_cache,subvol=__active/ROOT 0 0
UUID=...				/home		btrfs	rw,nodev,nosuid,relatime,ssd,discard,space_cache,subvol=__active/home 0 0
UUID=...				/opt		btrfs	rw,nodev,nosuid,relatime,ssd,discard,space_cache,subvol=__active/opt 0 0
UUID=...				/var		btrfs	rw,nodev,nosuid,noexec,relatime,ssd,discard,space_cache,subvol=__active/var 0 0
UUID=...				/run/btrfs-root	btrfs	rw,nodev,nosuid,noexec,relatime,ssd,discard,space_cache 0 0
/run/btrfs-root/__active/ROOT/var/lib	/var/lib	none	bind 0 0

# /dev/sdb1 LABEL=EFI
UUID=...	/boot	vfat rw,nosuid,nodev,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro,discard 0 2

Be careful about the configuration of /var/lib. In fact, we need to mount the main BTRFS partition in order to access the directory that should be binded. Moreover, do not forget to check the line about the /boot partition.

We also need to create the directory where the root BTRFS partition will mount (see in the fstab).

$ mkdir -p /mnt/btrfs-active/run/btrfs-root

Configure the base system

You will now root from the newly installed base system with a arch-chroot. The configuration begins with mainly the same operations that we have done before.

$ echo "KEYMAP=fr-pc" > /etc/vconsole.conf
$ echo "FONT=Lat2-Terminus16" >> /etc/vconsole.conf
$ echo "fr_FR.UTF-8 UTF-8" > /etc/locale.gen
$ locale-gen
$ echo "LANG=fr_FR.UTF-8" > /etc/locale.conf
$ export LANG=fr_FR.UTF-8
$ ln -s /usr/share/zoneinfo/Europe/Paris /etc/localtime
$ hwclock --systohc --utc
$ echo "mypc" > /etc/hostname
$ passwd

Configure a wireless network connection

We will configure a default profile for a wireless connection with SSID mySSID and a WPA security. Then we will make this connection automatic with wpa_actiond. Begin to install wpa_actiond with the base package netctl. The dialog package allow us the use of wifi-menu to generate netctl profiles.

$ pacman -S netctl iw dialog wpa_actiond

Then create a netctl profile from an example file. You can either automatically generate a profile file with wifi-menu or use the manual configuration below.

$ cp /etc/netctl/examples/wireless-wpa-config /etc/netctl/profile

You should modify /etc/netctl/profile to make it look like the following.

Description='A wpa_supplicant configuration file based wireless connection'
Interface=wlp3s0
Connection=wireless
Security=wpa-config
WPAConfigFile='/etc/wpa_supplicant/wpa_supplicant.conf'
IP=dhcp

Do not forget to change the path to the wpa_supplicant.conf file which should be /etc/wpa_supplicant/wpa_supplicant.conf.

Now, we can configure the WPA configuration of our mySSID wifi network. To configure this network, we use wpa_passphrase. We first backup the default configuration file.

$ cp /etc/wpa_supplicant/wpa_supplicant.conf /etc/wpa_supplicant/wpa_supplicant.conf.example
$ wpa_passphrase "mySSID" "mypassword" > /etc/wpa_supplicant/wpa_supplicant.conf

Do not forget the double quotes, especially if the SSID contains spaces. You may edit the file and remove the password which appear in clear.

You can now enable this service.

$ systemctl enable netctl-auto@wlp3s0.service

Configure the initial ramdisk environment

To generate the initial ramdisk, because you run with BTRFS partitions, you should modify some hooks in the /etc/mkinitcpio.conf. We will add a hook for btrfs. For this new hook, we must install the package btrfs-progs.

Secondly, BTRFS does not have a robust check file system so if we have the fsck hook, it will display an error. We have 2 solutions:

We erase the fsck hook and there will not have file system check anymore, even if other partitions are not BTRFS

We can create a fake fsck.btrfs command

$ ln -s /usr/bin/true /usr/bin/fsck.btrfs
$ sed 's/^\(HOOKS=.*fsck\)\(.*$\)/\1 btrfs\2/g' -i mkinitcpio.conf
$ mkinitcpio -p linux

Install and configure the bootloader

I choose gummiboot bootloader which is a simple bootloader for EFI systems. First of all, you install it. Before installing it, you may mount the efivarfs.

$ mount -t efivarfs efivarfs /sys/firmware/efi/efivarfs
$ pacman -S gummiboot
$ gummiboot install

Now, we can configure the bootloader following the specifications propose by http://www.freedesktop.org (see the references). Before configuring it, we will get the following variable:

$ export MACHINE_ID=`cat /etc/machine-id`
$ export KERNEL_VERSION=`uname -r`
$ export OS_IDENTIFIER=`cat /etc/os-release | grep ID | awk -F '=' '{print $2}'`
$ export MACHINE_HARDWARE=`uname -m`
$ export OS_VERSION=`echo $KERNEL_VERSION.$OS_IDENTIFIER.$MACHINE_HARDWARE`
$ export PARTUUID=`ls -l /dev/disk/by-partuuid/ | grep sda2 | awk '{print $9}'`

Create an entry in /boot/loader/entries.

$ vi /boot/loader/entries/$MACHINE_ID-$OS_VERSION.conf

title		Arch Linux
version		<value of $OS_VERSION>
machine-id	<value of $MACHINE_ID>
options		root=PARTUUID=<value of $PARTUUID> rootflags=subvol=__active/ROOT rw
linux		/<value of $MACHINE_ID>/<value of $OS_VERSION>/linux
initrd		/<value of $MACHINE_ID>/<value of $OS_VERSION>/initrd

We may also configure the main configuration file /boot/loader/loader.conf.

timeout		3
default		<name of the entries conf file, without the '.conf'>

The default should be fine since it use any entry file which begin with the machine ID.

You should now create the directories and copy the images into them.

$ mkdir -p /boot/$MACHINE_ID/$OS_VERSION
$ cp /boot/vmlinuz-linux /boot/$MACHINE_ID/$OS_VERSION/linux
$ cp /boot/initramfs-linux.img /boot/$MACHINE_ID/$OS_VERSION/initrd

Rebooting the system

Before rebooting, you should define a root password, unmount all the mounted filesystems and quit the chroot environment.

$ umount /sys/firmware/efi/efivarfs
$ exit
$ umount -R /mnt/btrfs-active
$ umount -R /mnt/btrfs-root
$ reboot

Possible errors

Make the ramdisk environment

You may have the following messages during the creation of the ramdisk.

==> WARNING: Possibly missing firmware for module: aic94xx
==> WARNING: Possibly missing firmware for module: bfa
==> WARNING: Possibly missing firmware for module: smsmdtv

It is probably some missing firmware you may install later.

Gummiboot installation

When installing gummiboot with pacman, you may have the following error message.

$ pacman -S gummiboot
...
Copied /usr/lib/gummiboot/gummibootx64.efi to /boot/EFI/gummiboot/gummibootx64.efi
Copied /usr/lib/gummiboot/gummibootx64.efi to /boot/EFI/Boot/BOOTX64.efi
Failed to create EFI Boot variable entry: No such file or directory

If this is the case, you may try the following before trying to reinstall gummiboot.

$ mount -t efivarfs efivarfs /sys/firmware/efi/efivarfs
$ pacman -S gummiboot

When rebooting, crash into ramfs

You may crash into the ramfs when rebooting. This is probably because you forget to indicate to the kernel which BTRFS subvolume is the default one. You may choose one of the two following solutions (both are explained in the above configuration):

Give the right option to the kernel with rootflags (see above for details)
Make one of the BTRFS subvolume the default. You can list the subvolume with the following command.
```
$ btrfs subvolume list -atp /mnt/btrfs-root
```

You should have something like the following

ID	gen	parent	top level	path
--	---	------	---------	----
256	18	5	5		__active/ROOT
257	7	5	5		__active/home
258	8	5	5		__active/opt
259	18	5	5		__active/var

We will now set the default subvolume which will be used by the kernel to launch the system. Keep the ID of the __active/ROOT subvolume and use it in the following command.

$ btrfs subvolume set-default 256 /mnt/btrfs-root/__active/ROOT

Be aware that this solution will fail to mount the root BTRFS system since it will mount instead this default subvolume. You may avoid it in the previous configuration because the root BTRFS file system must be mount in order to bind /var/lib.

References

For this article, there is two main references. The first one is the Beginner’s guide for Arch Linux installation. There is a lot of information on how to install an Arch Linux distribution and a lot of links to explain the details.

The second reference is mainly about the BTRFS configuration suggest by Fabio Mancinelli in his article.

In the rabbit hole

Don’t be lazy

Make it look like rainbows 🌈

What can we do to improve the reporting?

Colors in the terminal

Let’s make it green and red

First iteration

Second iteration

Third iteration

Problems to address later

Assertion reporting

Little recap

Storing the results

Failing again

More information in the report

Some notes

Equality assertions

Equality for other types

What about chaining?

Construct an Assertion

assert_that as a function

assert_that as a macro

Implementing is_equal_to

Conclusion

`cargo-sonar`

Announcing cargo-sonar

Why?

How?

Setting up cargo-sonar

Generate clippy output

Convert into Sonar generic issues

Push the generic issues to sonarcloud.io

What’s more?

cargo-audit

cargo-deny

cargo-outdated

What’s next

Dynamic iterators

Introduction

First try: return an Iterator type

Second try: associated type

Third try: impl Iterator

Fourth try: Box<dyn Iterator>

Fifth try: wrapping Box<dyn Iterator>

Conclusion

Take second try and third try, shake, and here we go

Fluent assertions for numbers in Rust

Introduction

API driven design

Chaining assertions

Owned versus referenced value

Constraints

Implementation

Testing the implementation

Conclusion

“Is not equal” and friends

More functions

Equality for other types

Naming test

Feedbacks

vim running Rust tests

Introduction

Selecting tests with cargo

Let’s do it in vim

Conclusion

Hole is back

Introduction

What changed?

Javascript

Docker

Java

Rust

Others

What else?

Conclusion

Rust testing ecosystem

Introduction

Testing in Rust language

Assertion

difference

Construct an `Assertion`

`assert_that` as a function

`assert_that` as a macro

Implementing `is_equal_to`

Announcing `cargo-sonar`

Setting up `cargo-sonar`

Generate `clippy` output

`cargo-audit`

`cargo-deny`

`cargo-outdated`

First try: return an `Iterator` type

Third try: `impl Iterator`

Fourth try: `Box<dyn Iterator>`

Fifth try: wrapping `Box<dyn Iterator>`

Selecting tests with `cargo`

Brief introduction to `sed`

`sed` with a script file

`BLOB` and `BOOLEAN` adptations

Create the `movim` user

Create the `movim` database

Get `????????????` in the `App Manager`