As usual it asks whether the receive location was started (it was) and if the isolated adapter runs under an account that has access to the BizTalk databases (which it does).
I had already followed the step to add the non-HTTP protocols and configured the web site to support net.tcp. I also configured the virtual directory to support http and net.tcp protocols.
To recap when creating the Receive Location I used the WCF-CustomIsolated adapter. I set the Binding to netTcpBinding and set the security mode to None to match the Anonymous access setting in IIS.
The problem was with the address Uri. I was using the address of net.tcp://localhost:808/path/service.svc which is shown in the examples. Changing the address Uri to /path/service.svc solved the problem.
Now when openeing the service in IIS I was able to see the usual service screen (using the http address) and view the wsdl. BTW, I wasn't able to get this to work with the WCF-NetTcp adapter because that too insists on the net.tcp:// address format.
But the next problem came when trying to create a client application to send messages to the net.tcp endpoint. I created a console application in Visual Studio but could not add the Service reference because it would not recognise the wsdl. So I switched the endpoint back to basicHttp and created the service reference in my console application and checked that it worked with basicHttp.
So then I switched the Receive Location back to net.tcp and changed the bindings and client endpoint in my console app to use net.tcp. I thought that was it but when I sent the message it would just timeout.
That's when I looked at the settings on the Receive Location again and realised maxConnections was 0 and the ReadQuota values were all zero.
These are what I set in the app.config of my client
<binding listenbacklog="16" maxconnections="16">
I also added the security section to match that of the server.
Finally that did the trick and I was able to submit messages with net.tcp.