libssh2_hostkey_hash man page
libssh2_hostkey_hash — return a hash of the remote host's key
const char * libssh2_hostkey_hash(LIBSSH2_SESSION *session, int hash_type);
session - Session instance as returned by libssh2_session_init_ex(3)
hash_type - One of: LIBSSH2_HOSTKEY_HASH_MD5 or LIBSSH2_HOSTKEY_HASH_SHA1.
Returns the computed digest of the remote system's hostkey. The length of the returned string is hash_type specific (e.g. 16 bytes for MD5, 20 bytes for SHA1).
Computed hostkey hash value, or NULL if the information is not available (either the session has not yet been started up, or the requested hash algorithm was not available). The hash consists of raw binary bytes, not hex digits, so it is not directly printable.