Roozky's Wonders

This is a list of cURL commands I found myself using regularly. More examples with deeper explanation can be found here

If you find yourself sitting behind a proxy

If you find yourself on a computer that is sitting behind a proxy:

curl --proxy http://proxy.example.org:4321 http://remote.example.org/

Form submit with data url-encoded manually

Simulates a form submission by POST with application/x-www-form-urlencoded content type and data url-encoded by you.

curl -d "year=1905&name=You+Me" http://www.example.com/form-submit

Form submit with data url-encoding automatically

Simulates a form submission by POST with application/x-www-form-urlencoded content type and data url-encoded by curl itself.

curl --data-urlencode "name=Joe Me" http://www.example.com/form-submit

POST with JSON body

curl -H "Content-Type: application/json" -d "{'happy': 'yes'}" http://www.example.com/json

Take a body from stdin

echo '{"text": "Hello **world**!"}' | curl -d @- https://api.github.com/markdown

Output HTTP status only

curl  http://www.google.com -sL -w "%{http_code}\\n" -o /dev/null

The key flat here is -w which prints the report using a custom format. Here is a list of available custom variables:

url_effective
http_code
http_connect
time_total
time_namelookup
time_connect
time_pretransfer
time_redirect
time_starttransfer
size_download
size_upload
size_header
size_request
speed_download
speed_upload
content_type
num_connects
num_redirects
ftp_entry_path

Check status of multiple URLs

for i in gumtree bbc; do curl  http://www.$i.com -sL -w "%{http_code} %{url_effective}\\n" -o /dev/null; done

Outputs:

200 http://www.gumtree.com/
200 http://www.bbc.com/

Recently I have faced a problem where I was asked to stop serializing selected object’s properties into JSON (to protect sensible information from being logged in json style logs). Code was using Jackson library which has lovely @JsonIgnore annotation specially designed for this type of problems. It almost looked too easy but luckily it was bit more complicated and had a change to learn more about Jackson internals.

Issue was that the object was used as a rest api request so I couldn’t just ignore a field as it will invalidate the request. At the same time it was being logged into the application log in the json format (json is easier to feed to Logstash) and we didn’t wanted the field to be logged as it contains information not suitable for logging.

During a search for a solution I had stumble upon @JsonView which looked promising but unfortunately it wasn’t working as it works as inclusion(you tell it which property you want) but I needed exclusion(you tell it which property you don’t want).

At the I was forced to extend a few Jackson classes and configure a mapper to use them.

Firstly I had to to create a custom property writer which gave me full control over property serialization so I could replace real property value with ‘*’.

public class SensitivePropertyWriter extends BeanPropertyWriter {
    private final BeanPropertyWriter writer;

    public SensitivePropertyWriter(BeanPropertyWriter writer) {
        super(writer);
        this.writer = writer;
    }

    @Override
    public void serializeAsField(Object bean,
                                 JsonGenerator gen,
                                 SerializerProvider prov) throws Exception {
        Object value = writer.get(bean);
        if (value != null && value instanceof String) {
            String strValue = (String) value;
            gen.writeStringField(writer.getName(), StringUtils.repeat("*", strValue.length()));
        }
    }

    @Override
    public BeanPropertyWriter withSerializer(JsonSerializer<Object> ser) {
        return this;
    }
}

The next step was to write a custom bean modifier in which I could configure custom property write on property by property base. I have just used regexp to match potentially sensitive fields but you can always go a step further and create an annotation.

import org.codehaus.jackson.map.SerializationConfig;
import org.codehaus.jackson.map.introspect.BasicBeanDescription;
import org.codehaus.jackson.map.ser.BeanPropertyWriter;
import org.codehaus.jackson.map.ser.BeanSerializerModifier;

import java.util.List;
import java.util.regex.Pattern;

public class SensitivePropertySerializerModifier extends BeanSerializerModifier {
    private static final Pattern PASSWORD_PATTERN = Pattern.compile("(?i)password");

    @Override
    public List<BeanPropertyWriter> changeProperties(SerializationConfig config,
                                                     BasicBeanDescription beanDesc,
                                                     List<BeanPropertyWriter> beanProperties) {
        for (int i = 0; i < beanProperties.size(); i++) {
            BeanPropertyWriter writer = beanProperties.get(i);
            if (PASSWORD_PATTERN.matcher(writer.getName()).find()) {
                beanProperties.set(i, new SensitivePropertyWriter(writer));
            }
        }
        return beanProperties;
    }
}

The last step was to configure ObjectMapper to use our new serializer modifier and test it.

import org.codehaus.jackson.map.ObjectMapper;
import org.codehaus.jackson.map.SerializationConfig;
import org.codehaus.jackson.map.SerializerFactory;
import org.codehaus.jackson.map.ser.BeanSerializerFactory;
import org.junit.Test;

import java.io.IOException;

import static org.fest.assertions.api.Assertions.assertThat;

public class SensitivePropertySerializerModifierTest {
    private ObjectMapper mapper = createMapper();

    public ObjectMapper createMapper() {
        SensitivePropertySerializerModifier modifier = new SensitivePropertySerializerModifier();
        SerializerFactory sf = BeanSerializerFactory.instance.withSerializerModifier(modifier);

        ObjectMapper mapper = new ObjectMapper();
        mapper.setSerializerFactory(sf);

        return mapper;
    }

    @Test
    public void shouldMaskPasswordFields() throws IOException {
        // given
        mapper.enable(SerializationConfig.Feature.SORT_PROPERTIES_ALPHABETICALLY);

        // when
        String json = mapper.writeValueAsString(new TestObject("password123"));

        // then
        assertThat(json).isEqualTo("{\"password\":\"***********\"}");
    }

    public static class TestObject {
        private String password;

        public TestObject() {
        }

        public TestObject(String password) {
            this.password = password;
        }

        public String getPassword() {
            return password;
        }

        public void setPassword(String password) {
            this.password = password;
        }
    }
}